C-KERMIT CHANGE LOG (Changes since 7.0.197 of 8 February 2000) Wed Dec 12 09:27:41 2001 ---7.0.197--- EVERYTHING FROM HERE DOWN was done after ckermit2.{txt,html} was last updated. FreeBSD 4.0 came out 16 March 2000. Building C-Kermit under it resulted in a version that lost CBREAK mode after return from curses. Adding -DCK_WREFRESH didn't fix it, as it does on other platforms where this happens. Building with curses instead of ncurses didn't fix it because in FreeBSD 4.0 curses *is* ncurses. It's the old curses-changes-console-buffering-but-endwin()- doesn't-restore-it problem. The only way around it is to re-enable the old setbuf(stdin,NULL) code in concb(). By Murphy's Law, there was no way to do this with the #ifdefs that were already there, so I had to add a new one: #ifdef NONOSETBUF. makefile, ckutio.c, ckuver.h, 16 March 2000. For Interactive UNIX System V/386 R3.2 V4.1.1, had to #ifdef out the #include of in ckcnet.c to avoid duplicate declarations of hostent, etc. Also added -DNOREALPATH to is5r3* targets, makefile. 19 Mar 2000. The binary produced for Interactive was not portable to generic SV/386 R3.2 because it contained calls to dup2(). Made a new target, sys5r3is, that builds on Interactive and produces a binary with no dup2() calls (by adding NOFDZERO, NOREDIRECT, NOZEXEC). This required removing the #ifndef NONAWS around extern int tt_rows, ttcols; in ckutio.c, which is safe, since the global declarations in ckcmai.c are not #ifdef'd. 20 Mar 2000. ---7.0.198---1.1.20--- 16 Apr 2000, the big Jeff/Frank code reconciliation. From small to big: . ck_ssl.h - Took Jeff's. . ckcpro.w - Fix a comment that caused problems with some compilers. . ckcker.h - Take Jeff's (some #ifdefs for PWDBUF size). . ckuver.h - Fix typos in comments, add FreeBSD 4.0. . ckcuni.c - Hebrew-7 fix. . ckwart.c - Copyright, improved debugging. . ckclib.c - makestr() bulletproofing and debugging. . ckufio.c - Debug / buffer-initialization for zinfill(). . ckuath.c - Took Jeff's. . ckucns.c - Some changes to select() setup. . ckupty.c - Debugging added. . ckuusr.h - Fix VT102 oversight, add security stuff, fix delmac() prototype. . ckuusx.c - Improvements to session logging, debugging, . ckuus6.c - Update delmac() calls, fix some text messages. . ckcmai.c - Update version numbers & dates, add some OS2 text patterns, initialize zinbuffer & zoutbuffer to NULL. . ckucmd.c - Better handling of directory-name completion. . ckuus4.c - delmac(), K5/FwdX msgs, Telnet debugging, . ckucon.c - no change (didn't tackle BEBOX stuff). . ckuusy.c - IKSD for K95, improved detection of -s with no files to send. . ckuusr.c - Make some TELNET keywords visible, fix some glitches, don't look for infodir in OS2. . ckcdeb.h - Shuffle lots o' #ifdefs. . ckudia.c - Fixes to ANSWER, etc. . ckuus5.c - K95 IKSD, fixes to Win95 short names, fix for local variable names that are prefixes of longer global names, add parameter to delmac() that says whether it requires exact name match, make IBM 3151 termtype visible, . ckctel.h - Took Jeff's. . ckuus2.c - New HELP KVERB + lots of new or updated help text. . ckuus3.c - Fix SET DIAL METHOD, update Telnet commands, fix SET PRINTER, . ckutio.c - 100 little things + make ttinl() ignore non-packet incoming when sending and streaming. . makefile - Jeff's secure-entry additions and changes reconciled with all my 7.0.197 changes. . ckuus7.c - Make IBM3151 visible, fixes for download directory selection, ensure no K95 popups vs startflags, updates to SHOW AUTH, . ckctel.c - Took Jeff's but added some casts to squelch compiler warnings. . ckcnet.c - Toof Jeff's but added #ifdef around #include for Interactive UNIX SVR3 from my copy and fixed some ANSI compiler assumptions. . ckuath.c - Took Jeff's. Checked out Jeff's new directory completion code. It's great, except for the case where the string matches exactly one directory that has no subdirectories. In this case Kermit should do full completion rather than partial. Added a rather crude hack to take care of this; it works and does not seem to have any bad side effects, but it's kind of scary anyway -- it does another nzxpand() (which should be OK since no znext()s follow), and it executes a GOTO into a parallel block of code, which from my memory of compiler construction might be expecting too much of some compilers. But it works fine with gcc and dumb old SunOS cc. ckucmd.c, 16 Apr 2000. Will look at BEBOX later, after we have BeOS 5.0 installed on a PC. Meanwhile, keep BeOS 4.5 up on BeBox so we can still support both. From Jeff: fix -M command-line option. ckuus[4y].c, ckcnet.c, 17 Apr 2000. Noticed that "help ." was rather unhelpful, because internally we recycle the XXDEF command code for it, so it just prints the HELP DEFINE text, which is not appropriate. I added a new variable, char * hlptok, which contains the string they are actually requesting help for, so we can disambiguate requests like this, and added new help text for ".". ckuus[r2].c, 17 Apr 2000. Added DIR /SUMMARY, which just prints the number of files and total size. ckuusr.h, ckuus[26].c, 17 Apr 2000. Added IF KBHIT. ckuusr.h, ckuus[26].c, 17 Apr 2000. Discovered, however, that "if kbhit getc %c" didn't get the character that caused IF KBHIT to succeed. Why? An intervening call to concb() has the side effect of clearing the console buffer, at least in SunOS (BSD). Which also explains why command typeahead doesn't work either -- concb() is called at the beginning of each command parse. But concb() doesn't clear the buffer in Solaris (System V) or Linux (POSIX). So it looks like the BSD/V7 version needs attention: (a) don't call concb() if the state didn't change from last time (messy); or (b) ckutio.c should remember the state and concb()/conbin() etc should do nothing if state didn't change; or (c) concb()/conbin() have to read in any pending characters first and stash them for later requests before calling stty() to change modes, and then conin()/conchk() and any other console input routines have to check the stash first. The question is: what to do about this? (a) gives the best performance boost, but requires changing code everywhere and anyway is not reliable since we will still lose characters when the state actually changes. (b) is like (a) except without changing code everywhere (and without the performance boost). (c) is a whole big deal, but still not reliable since there is a window between the time we read the characters and do the stty() or ioctl() to change modes. I implemented (b), but it doesn't help because in the test case, we actually do change modes (GETC calls conbin()). At least it saves us some redundant system calls. So I implemented (c) too. ckutio.c, 17 Apr 2000. Note: The (c) implementation is only for BSD/V7. It shouldn't make any functional difference elsewhere, but it might make a performance difference. Is it worth the risk? On the other hand, if we always did this, it would catch any other oddball platforms we might not know about. We can revisit... search for "conbuf". Speaking of jumping into the middle of a block, I checked to see if Kermit lets you do that. It shouldn't but it does, e.g.: echo ONE goto xxx if true { echo TWO :xxx echo THREE } echo blah :xxx echo FOUR Why? The GOTO code is calling fread() directly. Instead it should be calling getnct() (Get Next Command from TAKe-file). Fixed in dogoto(): ckuus6.c, 17 Apr 2000. When defining a variable or macro (with addmac()), we always call delmac() and then allocate new space for the new value. But we don't need to do this if the new value is no longer than the old one; we can just copy the new value over the old one. This could make certain loops run a lot faster. But a look at the code shows that it's just as hard to find out the length of the current value as it is to do what we do now. But let's see if it makes a difference with a loop that replaces two variables 10,000 times each: echo \v(time) for \%i 1 10000 1 { asg \%m \%i } echo \v(time) Trials with and without the purported optimization on an unloaded (but slow) Linux PC show no significant difference in elapsed time: about 84 sec. Obviously the bottleneck is elsewhere. I backed off on the change (see USE_VARLEN in ckuus5.c) and built a profiling version (linuxp) and ran the loop again. Results: % cumulative self self total time seconds seconds calls ms/call ms/call name 25.67 8.86 8.86 70045 0.13 0.25 lookup 24.43 17.29 8.43 9774786 0.00 0.00 ckstrcmp 9.07 20.42 3.13 190082 0.02 0.03 gtword 7.16 22.89 2.47 6374111 0.00 0.00 dodebug 5.01 24.62 1.73 350133 0.00 0.00 setatm 4.03 26.01 1.39 140068 0.01 0.01 zzstring 3.91 27.36 1.35 3 450.00 11483.16 parser lookup() can call ckstrcmp() as many as 3 times per table entry. So we should be able to get a big boost out of fixing lookup() not to do string comparisons if it doesn't have to. It certainly doesn't have to if the first letter of the target is not the same as the first letter of the table entry (caseless). This brings the runtime down to 66 seconds: 32.76 7.95 7.95 70045 0.11 0.12 lookup 10.47 10.49 2.54 190082 0.01 0.02 gtword 8.90 12.65 2.16 6374099 0.00 0.00 dodebug 5.97 14.10 1.45 350133 0.00 0.00 setatm 5.64 15.47 1.37 3 456.67 8069.83 parser That's a 27% improvement -- not bad for five minutes work. Next: what is dodebug() doing up there at number 3? Hmmm I thought I had defined IFDEBUG if BIGBUFOK but apparently not. It seems the debug() macro was defined in ckcdeb.h before defining BIGBUFOK. Moved the debug() definition to the right place and rebuilt. Executable size (on Linux, with profiling) before: 1481306; after: 1516266, a 34K increase. Now the loop executes in 56 seconds: 32.90 7.40 7.40 70045 0.11 0.11 lookup 11.38 9.96 2.56 190082 0.01 0.02 gtword 7.56 11.66 1.70 3 566.67 7479.84 parser That's a 50% speedup. The next thing would be to add a lookup() cache, which I started to do but didn't have time to finish. ckucmd.c, 17 Apr 2000. New files from Jeff: ckctel.c ckcmai.c ckuath.c ckuusy.c, implementing the options that make up the secure Telnet personality command line. 17 Apr 2000. Cleaned up ckuusy.c formatting and added a needed prototyp for dotnarg(). 18 Arp 2000. Back to the lookup cache... I can't think of a good (portable, efficient) way to maintain a dynamic cache. The question is when to replace an existing item by a new one. This can be done only with some combination of hit count and timestamp. But a timestamp requires a nonportable high-overhead system call, and the format of the timestamp is nonportable too. So let's try a static cache. By running a few nontrivial script programs and having lookup() collect statistics, and then running them through our all-purpose associative array builder, we find a short list of items that consistently get the most hits (IF, NOT, GOTO, etc), so we can build our cache from these when Kermit starts. Here are the top ones, with counts from a typical run: if 602 = 106 not 247 lit 97 cmdlevel 241 do 83 goto 231 _getargs 82 > 230 < 79 incr 173 _putargs 77 def 172 asg 69 _assign 159 xecho 59 echo 120 else 56 eval 112 Adding a 16-element cache results in not much speedup: 53 seconds, down from 56, even though the cache hit rate was over 70%. Reducing the size to 8 lowers the hit rate to 57% but does not lower the elapsed time. Reducing the cache to 4 gives a hit rate of 28% and increases the elapsed time to 55 seconds. So there is no special benefit to reducing the cache size. Some hand-tuning of the cache-search loop brings elapsed time down to 52 sec with the original cache of 16, and: % cumulative self self total time seconds seconds calls ms/call ms/call name 25.16 4.69 4.69 70065 0.07 0.07 lookup 14.91 7.47 2.78 190084 0.01 0.02 gtword 7.24 8.82 1.35 140071 0.01 0.01 zzstring 6.76 10.08 1.26 350142 0.00 0.00 setatm 6.28 11.25 1.17 60024 0.02 0.15 docmd 5.58 12.29 1.04 3 346.67 6192.79 parser This still beats the non-cache version by 8-10%, so let's keep it. Anyway, it's demonstrably a better lookup() than the one that has all the other improvements but no caching: 33.48 7.65 7.65 70048 0.11 0.12 lookup 11.55 10.29 2.64 190084 0.01 0.02 gtword 7.61 12.03 1.74 60024 0.03 0.16 docmd 5.43 13.27 1.24 350142 0.00 0.00 setatm 5.08 14.43 1.16 3 386.67 7603.16 parser Execution of the script goes down to 46 sec with no profiling, an 83% speed improvement. The lookup() caching code is selected by USE_LUCACHE, which is defined in ckuusr.h if NOLUCACHE is not defined. The cache is initialized in ckuus5.c, and searched in lookup() in ckucmd.c. To get the cache hit rate, do whatever you want in C-Kermit, and then "log debug" and "exit", then look for "cache" in the debug log. 18 Apr 2000. Giving a BYE command to a Kermit program that's in remote mode never does anything useful, but can easily do something bad. Added a test for this to the BYE code in ckuusr.c, 18 Apr 2000. EXIT (QUIT) could, under certain circumstances (which I can't reproduce but can still see how it might have happened), fail to return the saved-up exit status. Changed the code to put ckitoa(xitsta) in the cmnum() default field instead of using the old hokey way of doing it. Also fixed the HELP EXIT message which never mentioned the new text argument. ckuusr.c, 18 Apr 2000. Changes from Jeff to allow a NOLOCAL build of K95. ckctel.c ckuus3.c ckcdeb.h ckuath.c ckuus4.c ckuus6.c ckuus7.c, 18 Apr 2000. From Jeff: More changes for K95 IKSD. ckuus4.c ckctel.c ckuus5.c ckcnet.c ckuus3.c ckuath.c ckuus7.c ckuus6.c ckwart.c ckcdeb.h ckcpro.w ckuusr.c ckuusx.c ckuscr.c. 20 Apr 2000. Fixed long lines and trailing blanks in all source files. 20 Apr 2000. Added STREAMING notation to CRT and SERIAL displays. ckuusx.c, 20 Apr 2000. Added IF KERBANG. ckuusr.h, ckuus[26].c, 20 Apr 2000. From Jeff, changes mostly to avoid confusion when trying to use Zmodem with IKSD: ckuusr.c ckcdeb.h ckuus4.c ckcpro.w ckuath.c ckudia.c ckuusy.c ckuus6.c ckuscr.c ckuus2.c ckuusx.c, 22 Apr 2000. Filled in a skeleton for user-mode chroot, so far just the parsing and setting. Still have to fill in the actions for all zblah() routines (about 24 of them) that take filenames as arguments, which is not so bad, but then we also have to worry about every fopen() call in the mainline code. I'm not sure this is really practical the way the code is now structured; even if we cover every single call, we'll forget when adding new ones. And we can't really front-end fopen() by yet another z...() routine (really, a whole bunch of them) without changing all the ck*fio.c modules. Maybe I'll have to give up on this (again) (or as Jeff suggests, make an fopen() replacement like we did for printf, etc). Files touched today: ckcdeb.h, ckuusr.h, ckuusr.c, ckufio.c. Everything is in #ifdef CKROOT. 22 Apr 2000. Discovered something wrong with cmifi2() when parsing a wildcard that expands to a list of file or directory names, if it is given the name of a directory. If the directory has no subdirectory, it matches nothing. If the directory has one or more subdirectories, it matches directory/first-subdir. The same is true when calling cmifi2() to match only directory names (as opposed to cmdir(), which matches only one name). Example: "dir /dir subdir". The problem is quite deep in traverse(), which I swore never to touch again without a total rewrite, so I added an ex-post-facto fixup to nzxpand(): if the diractive flag is not set (a pre-existing disgusting hack) and zxpand has been given a not-wild directory name, it should just return it. However, this was rather tricky since we had to fake all the fgen()/traverse()/addresult() data structures, even when they had not been initialized yet. A fundamental change that needs watching... ckufio.c, 23 Apr 2000. Even with this fixup, it is not (and never has been) possible to get a directory listing of a single directory file if that directory contains any files. But with the above fix, it's now possible to overload the /NORECURSIVE DIR switch to force this: "dir subdir" lists all the files in subdir, but "dir /norecursive subdir" lists subdir itself. ckuus6.c, 23 Apr 2000. Discovered the code in UNIX addresult() to append a directory separator to the end of a directory name if one wasn't there already never worked (this was masked by higher levels of code and not noticeable until the fix just above). Fixed in ckufio.c, 23 Apr 2000. Discovered ckradix() did not fail when given illegal input. Fixed in ckclib.c, 23 Apr 2000. Changed cmnum() to allow specification of octal radix. Fixed cmfdb() to allow passing of radix to cmnum(). This could be easily extended to any radix 2-36. ckucmd.c, 23 Apr 2000. Noticed that cmifi() listed directories in its "?" file list even when files-only is specified, even though it actually returns only regular files once the field is terminated. Fixed in ckucmd.c, 23 Apr 2000. Added CHMOD command for UNIX only. Operates on single or multiple files or directories, or any mixture. Accepts the normal assortment of switches: files-only, directories-only, recursive, verbose, page, etc. HELP CHMOD for the list. The permission code is strictly an octal number, no "chmod g+x" etc, although it would not be hard to add if anybody cared. ckuusr.[ch], ckuus[23x].c, 23 Apr 2000. To extend this to other operating systems requires a more general design: SET FILE PERMISSIONS (or PROTECTION) { /UNIX, /VMS, ... } (plus the other switches of course), which tells Kermit how to interpret the code, the default being either the native format or some new portable format that we devise. Implementation of the client/server version of this is a big deal since it has to account for the semantics of every known file system, not just the simplistic one used by Unix. For example, in VMS we have four categories of user, not just three, and we have delete permission in addition to RWE. Execute permission has different meaning on different platforms, and sometimes even on the same platform (e.g. directory search). VMS, AOS/VS, and NT have ACLs, etc etc. A protocol must be designed that accounts for everything, and also is extensible to accommodate future developments (or discoveries). Kermit's current model (as used in A packets) is totally inadequate. Also this begs the question of "wildcards" in the cross-platform client/server environment. We've always required that client commands understand the server's wildcard syntax, which has its good and bad points. But if we don't want to allow platform-specific protection codes in the protocol, why should we allow platform-specific wildcard syntax? Worse, how could we add REMOTE SET FILE PROTECTION commands that mixed the two? From Jeff: SET IKS commands (executable only in IKSD.CONF): set iks anonymous { on, off } set iks bannerfile set iks cdfile set iks cdmessage { on, off } set iks helpfile set iks initfile set iks userfile set iks xferlog { on, off } set iks xferfile Also: . Added "ktelnet" as synonym for I_AM_TELNET . changed --privid to be Unix only. ckcmai.c ckuus[r356y].[ch], 24 Apr 2000. Some debug logs showed that isdir() is often called twice in a row on the same file. Rather than try to sort out clients, I added a 1-element cache to Unix isdir(). ckufio.c, 24 Apr 2000. Another refinement to directory-name completion: "cd bl" completes to "blah/ " if there is only one directory name that starts with "bl" and it contains no subdirectories. However, "chmod 664 bl" completes to "blah/" and then beeps. Why? Because in this case we are parsing either a filename OR a directory name, whereas CD parses only directory names. So the real test for whether to beep is: in the directory-only case, beep if the directory contains subdirectories; in the directory-or-file case, beep if the directory contains any files at all; otherwise complete and add a space. cmifi2(): ckucmd.c, 24 Apr 2000. More from Jeff: ON_LOGIN macro for IKSD, make SHOW MACROS terminate lines correctly in IKSD. ckcmai.c, ckuus5.c, 24 Apr 2000. From Jeff: More SET IKSD commands, rlogin ID swap, etc. ckcnet.c, ckctel.c, ckuusy.c, 25 Apr 2000. More fooling around with user-level chroot. There is now an invisible CHROOT command, which sets the user's access root and CDs to it, done internally by zsetroot(). The zinroot() function checks to see if its argument is in the root, if any. zinroot() calls have been added to zopeni(), which is called by (e.g.) TYPE, TRANSMIT, and XLATE, and to zchdir() for CD. As a proof of concept, it seems pretty solid; it's like the Roach Motel. Next steps: . Put the checks everywhere they are needed in ckufio.c. . Make a front end for fopen() to be used in the mainline code. . Give appropriate error messages when root violations occur. . Don't show full paths, e.g. in zgtdir(), zfnqfp(), etc. . Consider allowing multiple roots rather than just one. ckufio.c, 25 Apr 2000. IKSD changes from Jeff, ck_ssl.c ckcmai.c ckcnet.c ckuus4.c ckuus5.c ckuusy.c, 26 Apr 2000 Got rid of homdir variable. It was totally unreliable since it was being set to whatever pointer zhome() happened to return at some time in the past, which might no longer be valid in the present, plus now with CHROOT it can change after C-Kermit starts. Replaced all references to homdir to zhome() calls. ckuus[57].c, 26 Apr 2000. Discovered that realpath() fails with ENOENT if given a wildcard, even though it does its job. Added a check for this. zfnqfp(): ckufio.c, 26 Apr 2000. Found lots of ckufio.c functions didn't check their string-pointer arguments for NULL. Fixed in ckufio.c, 26 Apr 2000. Discovered that if zxpand() returned 0 because of an argument check, it left all its counts and pointers intact from the previous time it was called. Fixed in ckufio.c, 26 Apr 2000. Added setroot checks to zopeno(), zchki(), zchko(), zdelet(), zrename(), zcopy(), zgperm(), zfcdat(), zstime(), zmail(), zprint(), zmkdir(), and zrmdir() (this was already in zchdir() and zopeni()). Made zhome() return the root directory. We now have a pretty solid access restriction mechanism that works as long as we go thru ckufio.c for file access. ckufio.c, 26 Apr 2000. What we don't have is a way to make the path left of the root invisible. Doing that would be a very big deal, involving conversion of every filename in every spot where they are referred to or displayed -- not worth the trouble or the risk. Also note that CHROOT has to set NOPUSH because there is no way it can control what the user does if she PUSHes, or gives a system command. This is another difference from system chroot. Still to do: . Make a front end for fopen() to be used in the mainline code. . Give appropriate error messages when root violations occur. . Consider allowing multiple roots rather than just one. Multiple roots are simple in theory -- just change zsetroot(), zinroot(), and the CHROOT command parser to account for them. But in practice this could be quite confusing -- which root is "home", etc. Plus in UNIX, this whole idea would be very foreign. Maybe it would make sense in Windows. Now for fopen()... The trick would be something like this in ckcdeb.h: #ifdef CKROOT #ifdef fopen #undef fopen #endif /* fopen */ #define fopen ckfopen /* and then a prototype for ckfopen */ #endif /* CKROOT */ and like this in ckufio.c: #ifdef CKROOT static FILE * ckfopen(name,mode) char * name, * mode; { if (!zinroot(name)) { debug(F110,"ckfopen setroot violation",name,0); return(0); } debug(F110,"ckfopen setroot ok",name,0); } #endif /* CKROOT */ But some experimentation shows that zinroot() always succeeds because C-Kermit never tries to open a file without first parsing its name, and the parse has already made the check thru zxpand(), zchko(), or other ckufio.c function. The only exception is for filenames hardwired in the code: the init file, the IKSD conf file, the fwdx_xauthfile. Do we care about these? Init file: No, because there would be no way to set the root before executing the init file anyway, not even if there was a chroot command-line option, since the init file is executed first (if the init file itself includes a CHROOT command, of course it will be effective for all subsequent commands in the init file). The IKSD conf file? No, because IKSD does a real (system) chroot(), so Kermit CHROOT is irrelevant. How about fwdx_xauthfile? Assuming we don't care about it either, we're done except for cosmetics, testing, and documentation. Added zgetroot() to return the root directory. ckufio.c, 27 Apr 2000. Added root display to SHOW FILE. ckuus4.c, 27 Apr 2000. Added SET ROOT (visible) as a more proper name for CHROOT (invisible). ckuusr.h, ckuus[r3].c, 27 Apr 2000. Made zinroot() set a global flag, ckrooterr, to its result. Higher level clients of zopeni(), zxpand(), etc, can check this flag to see if the error was a normal one (e.g. that sets errno) or a root violation, and tailor their error messages accordingly. ckufio.c, 27 Apr 2000. Added prototypes for new functions to ckcdeb.h and ckuusr.h, 27 Apr 2000. Fixed up most error messages. ckcker.h, ckuus[r7x].c, ckucmd.c, 27 Apr 2000. Added SET ROOT / CHROOT help text. ckuus2.c, 27 Apr 2000. "type" with no filename didn't give an error message. Fixed in ckuusr.c, 27 Apr 2000. A user noticed that the QNX 32-bit binary did not have BIGBUFOK set, and so couldn't handle large loops. Fixed in ckcdeb.h, makefile, 28 Apr 2000. From Jeff, 28 Apr 2000: . Fix problems with TYPE, askmore(), etc, while executing IKSD.CNF, ckuusx.c. . Enable CKROOT for K95. ckcdeb.h. . Add zinroot(init-file) checks. ckuus5.c. . Improved wording of some login error messages. ckcmai.c. . Some code-shuffling in ckufio.c. Reformatted Jeff's ckcmai.c messages since they were longer than 80 columns. 28 Apr 2000. Changed Jeff's changes to doinit() to not peek at private variables. There's no need; just call zinroot() on the filename. The checks to ckrootset, etc, are just micro-performance-enhancers. ckufio.c, ckuus5.c, 28 Apr 2000. From Jeff: SET IKS ANONYMOUS commands, including ROOT. ckuusy.c, 28 Apr 2000. Added PROMPT command. This is like DO or TAKE, but it enters a new command stack frame with the command source pointed at the keyboard. This is useful for debugging scripts. Use the END command to return to the previous command stack level. ckuusr.[ch], ckuus2.c, 28 Apr 2000. Fixed SHOW STACK and RETURN to allow for non-zero command levels to be PROMPT, rather than DO or TAKE. Thus either END or RETURN can be used to return from the prompt to the level that invoked it, with their usual effects (END 1, END 0, etc; RETURN ). STOP and EXIT have their usual effects too, as do Ctrl-C and TRACE. ckuus[r5].c, 28 Apr 2000. Changed VMS, AOS/VS, VOS zfnqfp's to set the len member of the returned struct to the actual length of the full pathname. For some reason I never bothered doing that before. The length was never used until the SET ROOT code needed it. ck[vdl]fio.c, 28 Apr 2000. Added -DPOSIX_CRTSCTS to Solaris entries 2.4 and later, since it appears a switch was made from the SVR4 hwfc API to the POSIX one, and reports indicate the SVR4 one doesn't work at all in Solaris 2.4 and later, but the POSIX one does. makefile, 28 Apr 2000. Added DELETE /SUMMARY (only lists the summary, no heading, no file lines). ckuusr.h, ckuus[26].c, 28 Apr 2000. Added a cast to rlog_ini() to squelch a compiler warning, ckcnet.c,28 Apr 2000. Changed PROMPT command to include optional text to print. ckuus[r2].c, 28 Apr 2000. (NOTE: might want to make it actually set the prompt.) Changed previous PROMPT command change to make the optional text be the prompt string itself, which goes on the stack, so that END/RETURN from this level restores the previous prompt automatically. This involved adding a new cmgetp() routine to the command module. ckucmd.[ch], ckuus[r2].c, 30 Apr 2000. Added \v(iprompt), the current interactive prompt string ("\v(prompt)" was already used by SET LOGIN). ckuusr.h, ckuus4.c, 30 Apr 2000. HELP FUNCTION blah was messed up for the last 4 or 5 functions because break; statements were missing from the switch(). ckuus2.c, 30 Apr 2000. Jeff: SHOW CONNECTION should not display encrypted status of a closed connection, since we don't know it any more. ckuus3.c, 1 May 2000. Discovered that CP1252 is erroneously equated with ISO Latin 1 in ckuxla.[hc]. I'll have to fix this later. Meanwhile, because of this, any loop that looked up the file character-set name from its index would return "cp1252" when the file character-set was Latin-1. I added a special case for this in TRANSLATE and SET TERMINAL LOCAL-CHARACTER-SET. Also, consolidated the various repetitive switch statements and loops that get the local display charset name into a new routine, getdcset(). ckuusr.[ch], ckuus7.c, 1 May 2000. Added TYPE /TRANSLATE-FROM:cset /TRANSLATE-TO:cset (Synonyms: /XLATE-FROM to /XLATE-TO, /CSIN and /CSOUT). By default no translation is done, as before. The default target character set, which is used if the /TRANSLATE-TO: switch is not given, or if is typed after the colon, is the one returned by getdcset(), which in Unix is the file charset, and in Windows is the console character set. Getting this to work (such as it does) wasn't easy, at least not without duplicating hundreds of lines of hairy code from xlate(). The object was to find a clever way to reuse (and not touch) existing code, and I seem to have done this in a roundabout way. Still, the result is far from perfect: . For sanity, this code is included only if UNICODE is defined. . /MATCH and /WIDTH might not work right, especially for multibyte sets. . You can't TYPE a UCS-2 file. It's also less than satisfying on Windows, since there is really no good way to know how to default the file character set: an OEM code page, a Windows code page, or UCS-2. Or even (in Win 2000) UTF-8. What does Microsoft do? They actually look at the contents of the file and decide what it is from statistics, yuk. ckuusr.[ch], ckuus[26].c, 1 May 2000. Adjusted TYPE command code to never assume that a buffer is NUL-terminated, or that the file to be typed does not contain NUL characters. This was another substantial rewrite of dotype(); no more [ck]str[n]cpy(), no more strlen(), no more printf(), etc. Replaced the printf() used to display each line with a call to a new function, typeline(), that can be filled in for each OS (it's currently blank for K95). I also made some adjustments for /WIDTH vs UCS-2. Plus now you can TYPE a UCS-2 file and see something, even if it's only gibberish (at least the ASCII comes out right). ckuus6.c, 2 May 2000. Changed the TYPE switch syntax. /CHARACTER-SET:name identifies the file's character set. /TRANSLATE-TO: identifies the target character set; in K95, this is ignored; the typeout() routine automatically converts to the appropriate screen character-set. Elsewhere, /TRANSLATE-TO: is used if given, and defaults to the current file character-set. ckuus[r2].c, 2 May 2000. Made sure that TYPE /MATCH works, at least for single-byte sets. It is applied after translation to the target set. There's no way to specify a UCS-2 pattern anyway. 2 May 2000. From Jeff: Better charset defaults for TYPE command in K95. ckuusr.h, ckuus[r6].c, 3 May 2000. Added code to handle TYPE'ing in K95 when the source charset is unknown. ckuus6.c, 3 May 2000. Discovered that "for \%i 1 \%n 1 { echo \frandom(4) }" printed 0 1 2 3 0 1 2 3 0 1 2 3, etc -- not random at all -- on SunOS but not on Linux or Solaris (didn't test elsewhere). Added an additional randomizer for this case. ckuus4.c, 4 May 2000. Discovered that COPY /SWAP didn't work right because of sign extension. Made it treat the characters as unsigned. Sigh. ckuus6.c, 5 May 2000. The problem with TYPE in K95 is definitely byte-order related. We have two global variables, byteorder and ucsorder. What are they? byteorder = machine's natural byte order. ucsorder = SET FILE UCS BYTE-ORDER value, supersedes byteorder. If no SET FILE UCS BYTE-ORDER command has been given, ucsorder takes the value of byteorder. But neither of these says what the byte order of the actual file is, in case it was switched because of the BOM, which supersedes both byteorder and ucsorder. So I added a new one: fileorder = byte order of current file. (Turns out this wasn't actually needed.) But there is still more confusion. In which order does xgnbyte() return its bytes when translating to UCS2? Bigendian: If filling packets (what == W_SEND) or if ucsorder == BE. Little endian: Otherwise. OK, so on the Sun (BE) TYPE gets the bytes back from xgnbyte() in BE order, and TYPE /CHAR:xxx works right for UCS-2 files that are either BE or LE, as well as for UTF-8 and all other character sets. But no TYPE /CHAR:xxx command works on Linux/Intel, which is Little Endian. Why? We send bytes to xpnbyte() in the order we receive them from xgnbyte(). But xpnbyte() assumes that incoming UCS-2 bytes are in BE order; in this case, they aren't. Evidently we must be taking some path through the code that was never taken before because all the other translation methods work (file transfer, TRANSLATE, and TRANSMIT). Finally, it turned out that byte-swapping was not applied consistently by xgnbyte(). There was one place where it let the end of line slip through without swapping, even though it was swapping all the other bytes, and this is what was clobbering TYPE, since it looks for the end-of-line. Fixed in ckcfns.c, 5 May 2000. All kinds of TYPE /CHAR: combinations were tested on both Sun (BE) and Linux (LE) successfully. However, it's possible that a last-minute byte-swap of the line buffer might be needed for K95 in typeline(). Fixed bad parser recovery from "type /char:". ckuusr.c, 5 May 2000. Jeff built new K95G and found that: > TYPE ckctel.h: > > . line counts are completely off but otherwise appears to display the > file correctly > The non-charset-aware part of typegetline() was looking for CRLF but K95 zchin() was never returning CR. Changed to look only for LF. > TYPE /char:utf8 utf-8-test.txt > . displays nothing > Debug log shows that typeline() is being given a buffer in proper UCS-2 Little Endian format. So this problem must be in the K95 section of typeline(). However, I did notice that there was junk at the end of the buffer and the length was an odd number (but this would not cause the void display). Fixed typegetline() to properly detect and lop CRLF, and fixed the debug(F011) format to produce a more useful result. > TYPE /char:utf8 ckctel.h > . displays garbage > As it should. > SET TERMINAL UNICODE OFF > > TYPE ckctel.h: > > . displays data as NUL char NUL char NUL char .... > I changed typegetline() to check tt_unicode. If set, it always translates to UCS-2. If not set, it translates to whatever outcs is. From Jeff: Fixes to K95 aspects of TYPE command. ckuus6.c, 7 May 2000. Tested latest K95G.EXE on Win95. It works for all cases except /CHAR:UCS2. In this case it looks like synchronization is lost on alternate lines. So the first line comes out OK, but the second line is appended to it as bunch of blobs, and so on through the file. Debug log shows the following buffer passed to typeline() (for the latin1.txt file): ISO 8859-1 L atin Alphabe t 1char de c col/row oc t hex descr iption Notice the with no intervening NUL, which puts the next line out of phase, and the trailing with no after it. A closer look at the debug log shows that in this case, the wrong (non-translating) section of typegetline() is being executed, probably because incs == outcs. So I rearranged the code to ensure that the translating section of typegetline() would be executed if the source character set is UCS-2. It works fine in Unix but I can't test it in K95 until the next build. ckuus6.c, 7 May 2000. Fixed some parsing glitches in TYPE /CHAR:xxx. ckuusr.c, 7 May 2000. Because the default character-set in C-Kermit is ASCII, TYPE /CHAR: tended to show lots of ?'s if you started C-Kermit without specifying a file character-set or /TRANSLATE-TO switch. Added a new environment variable, K_CHARSET. This sets the file character-set when Kermit starts up, before reading the init file (if it does). This allows the file character-set to be set even when the init file is not executed. ckuus5.c, 7 May 2000. Added SIGINT handler to TYPE, since it was liable to malloc arbitrary amounts of space that would never be freed if regular SIGINT trap was used. ckuus6.c, 7 May 2000. So for the record, here's how the character-set stuff in TYPE works: If you TYPE a file without giving a /CHARACTER-SET switch, it simply dumps the lines to the screen and all the other switches apply normally. If you give a /CHARACTER-SET:xxx switch, this tells Kermit that the file's character set is xxx, and it should translate it to the character-set specified in the /TRANSLATE-TO:yyy switch, or if none given, to the current file character set. Except that in K95, the /TRANSLATE-TO switch is ignored and the file is always translated to the code page appropriate to your screen. If the source character-set is UCS-2, then the byte order is determined as follows: 1. If the file begins with a Byte Order Mark (BOM), this determines the byte order; otherwise: 2. If a SET FILE UCS BYTE-ORDER command was given, the specified byte order is used; otherwise: 3. The natural byte order of the underlying computer hardware is used. If the target character-set is UCS-2, the byte order is the natural hardware byte order. File lines are identified based on the source-file character set. Switches such as /MATCH, however, are applied on the translated line. New K95G build from Jeff. TYPE /CHAR:UCS2 now works fine, except the line counts are bit off. Actually, this happens no matter what the character set, and even if /CHAR: not specified. It's not a factor-of-two thing either -- it stops at 16 lines on a 25-line screen. Debug log shows cmd_rows is correct (25). Problem: The askmore() code checks the length of a line to see if it will wrap. But if it's UCS-2, the length is double. Fixed in dotype(). Also noticed a potential problem in tab expansion versus UCS-2, also fixed in dotype(). ckuus6.c, 8 May 2000. From Jeff: HTTP CONNECT. Built-in SOCKS support for K95. ckcnet.[ch], ckcmai.c, ckuus[r5].c, 8 May 2000. From Jeff: Minor changes to HTTP CONNECT & SOCKS, SHOW FEATURES. ckcdeb.h, ckuusr.h, ckuus[235].c, ckuath.c, ckcnet.c, 9 May 2000. From Jeff: Change TYPE command not to show UCS BOM. ckuus[r6].c, 9 May 2000. From Jeff: HTTP proxy support, SHOW TCP command. ckcnet.c ckuus4.c ck_ssl.c ckuath.c ckuus2.c ckuus3.c ckuusr.c ck_ssl.h ckuusr.h ckcmai.c ckcdeb.h, ckuus5.c, 10 may 2000. Added a new routine: chkfil(name,flags). Given the name of a file, analyzes the first 4K of its contents to see if it is text or binary and, if text, whether it is 7-bit text, 8-bit text, UTF-8, UCS-2/LE, or UCS-2/BE. On a Sparc-20, this routine takes about 0.025 second to execute, primarily due to the file open/read/close sequence, which does not seem too painful. Can be built with or without Unicode support; if built without, simply tells whether the file is text or binary. ckuusr.h, ckuusx.c. 10 May 2000. If UNICODE is defined at compile time, call chkfil() from TYPE if /CHARACTER-SET was not specified. This allows TYPE to recognize and handle Unicode files automatically. Also made some corrections for the non-Unicode case. ckuusr.c, 10 May 2000. Cleaned up and fixed NOUNICODE build. ckuus6.c, 10 May 2000. chkfil() can have other uses too -- for example, it can be used instead of, or to supplement, filename patterns for SET TRANSFER MODE AUTOMATIC. In a way, it's better than patterns, so it really could replace them. But since patterns are already in use, this would be an incompatible change, so all we can do is call chkfil() if no patterns match. So if users want to skip the hokey name patterns and use chkfil() instead, they can simply SET FILE PATTERNS OFF and then not only will every file be sent in text or binary mode based on its actual contents, regardless of its name, but also Kermit will switch character sets automatically for text files: . If file is UTF-8, it switches the transfer charset to whatever is associated to UTF-8 -- if anything -- otherwise to the currently selected transfer character set. Ditto for UCS-2. . If the file is not UTF-8 or UCS-2, it switches back to the prevailing file and transfer character sets. . When receiving, and the transfer character set has suddenly switched to (say) UCS-2, the receiving Kermit handles this based on all its preexisting settings: SET UKNOWN CHARACTER-SET, its association for UCS-2 (if any), or else its current file character-set. Added code for all this to sfile(). ckcfns.c, 10 May 2000. As matters stand, there is no way to restore the C-Kermit 7.0 behavior for files whose names don't match any patterns. For this we need Yet Another SET Command: SET FILE INSPECTION { ON, OFF }, with its status displayed in SHOW FILE. ckcker.h, ckuus[247].c, 10 May 2000. Added a chkfil() call to DIRECTORY /XFERMODE, so now it really can show what mode each file would be transferred in. ckuus6.c, 10 May 2000. Suppose we have FILE PATTERNS ON, FILE INSPECTION ON, and FILE CHARACTER-SET LATIN1, and we "send *". Now suppose we have a UTF-8 or UCS-2 file with a text filetype like utf8.txt. The pattern code will switch to text mode, but not to UTF-8 -- it will misinterpret the file as Latin-1. So this means we should probably call chkfil() even if the pattern check succeeded and found text. For that matter, we should call chkfil() if it found binary too, in case the pattern was wrong or inappropriate (e.g. a VMS .COM file (text) on a Windows computer where .COM files are binary, or on a .DOC file that can be either plain text or MS Word). So in other words, we should always call chkfil(), which in turn makes "set file patterns" useless: . If patterns and chkfil() agree, we didn't need patterns. . If patterns and chkfil() disagree, chkfil() takes precedence. There really is no reason to keep patterns, but they are already documented and in use. What's the graceful way out of this? State that inspection takes precedence over patterns, so if inspection is on, patterns are ignored. If inspection is off but patterns are on, patterns are used. OK done: ckcmai.c, ckcfns.c, ckuus[457].c, 11 May 2000. (Actually, there is one benefit to patterns: users can change them, whereas they can't change the chkfil() algorithm except by changing the code.) Next problem: we need a new set of associations for when chkfil() finds a 7- or 8-bit text file and its current file character-set is not appropriate: . What is the default character-set for 7-bit text? . What is the default character-set for 8-bit text? And new commands for them: SET FILE DEFAULT 7BIT-CHARACTER-SET xxx SET FILE DEFAULT 8BIT-CHARACTER-SET xxx Done. The default default 7-bit character set is ASCII; the default default 8-bit one is the same as the default file character-set, which depends on the platform. ckcker.h, ckuxla.c, ckuus[27].c, 11 May 2000. SET FILE CHARACTER-SET also must set this: if the name given is that of a 7-bit set, this also sets the default 7-bit set; ditto for 8-bit. ckuus7.c, 11 May 2000. Added the default 7- and 8-bit sets to SHOW FILE and SHOW CHARACTER-SET. ckuus4.c, 11 May 2000. Now the real stuff: in sfile(), we now switch automatically to the default 7-bit set if chkfil() said FT_7BIT and to the default 8-bit set if it was FT_8BIT. This allows, for example, a mixture of binary, UCS-2, UTF-8, Latin-1, and German ISO 646 files to be sent in a single group. Obviously, however, we can't switch automatically between (say) Latin-1 and Latin-2, or German and Spanish. I didn't do anything special about Kanji -- if anybody notices, then I'll worry about it (I haven't heard a peep about Kanji conversions since 1994, when we first put them in, so I truly doubt if anybody is using them). ckcfns.c, 11 May 2000. Fixed a typo in HELP PURGE. ckuus2.c, 11 May 2000. Fixed a problem with FSEEK /LINE \%c LAST. It failed to work if it was already at or after the beginning of the last line. ckuus7.c, 11 May 2000. K95 TYPE is broken again. It can't handle a Big-Endian UCS-2 file. Furthermore, once I try to type such a file, then it can't type any other kind of file thereafter. It doesn't matter if the BE file has a BOM or not. The problem is that we set the ucsorder flag based on the chkfil() result (in case this file didn't have a BOM, so the subsequent open() and xgnbyte() sequence won't detect it). But ucsorder is a global and persistent flag. Remember: byteorder = native byte order for hardware (doesn't change) ucsorder = SET FILE UCS BYTE-ORDER value (only changed by user) fileorder = byte order detected for current file (detected per file) I was using ucsorder in some places where I should have been using fileorder. Plus some other mistakes. Fixed in ckuus[r6x].c, ckcfns.c, ckuxla.c, 12 May 2000. Tightened up the Ctrl-C handling in dotype(). ckuus6.c, 12 May 2000. Here, by the way, is a command file that sends a mixture of files (Unix): set xfer mode auto ; This is the default set xfer char latin1 ; Because TRANSPARENT is the default set file char german ; This sets the default for 7-bit text files set file char latin1 ; This sets the default for 8-bit text files assoc file ucs2 utf8 ; Because there is no association by default assoc file utf8 utf8 ; Ditto send * Everything works perfectly on the Sun, as usual, but in Windows we still can't type Big-Endian UCS-2 files (but, as usual, TRANSLATE works on the same files). But oddly enough, everything works fine on PC Linux, which is Little Endian. The difference is that in Linux, we are outputting in Latin-1, whereas in Windows, it's in UCS-2... Aha, here's the problem: we read the first two bytes of the file and put them, left to right, into a USHORT. Then we compare the USHORT with 0xFEFF and if equal, then we say the file order is 0, i.e. BE. If it's 0xFFFE, we say it's 1 (LE). The fallacy is that if we read FF and then FE on an LE machine, the USHORT will look like FEFF, not FFFE. And vice versa. So... Machine File Bytes USHORT Fileorder BE FE FF FEFF BE <-- Machine order BE FF FE FFFE LE LE FE FF FFFE BE LE FF FE FEFF LE <-- Machine order In other words, whenever the BOM, saved in a USHORT, is 0xFEFF, the file is in the machine's native byte order. Well, that sounds good, but it's not enough. Now K95G does the opposite of what it did before. After uncovering & fixing several layers of problems now all that's left is is xpnbyte(). What was wrong? We were outputting the bytes in fileorder rather than ucsorder. Remember: fileorder: applies to input file. Set by chkfil or xgnbyte, used by xgnbyte. ucsorder: applies to output file. Set by user & used by xpnbyte. (ucsorder is also used for input when we don't have BOM and detection is not active.) ucsorder is the same as byteorder by default, but can be changed by the user in case they want to write out a file opposite to the machine's natural order. Of course that wouldn't make much sense when TYPEing... But never let it be said we didn't let users do what they want. Files changed: ckuus6.c, ckcfns.c, ckuxla.c, 12 May 2000. To do: . Clean out most of the debug calls. . Check file transfer. . Uncouple CP1252 from Latin-1. . Maybe add an ANALYZE command. From Jeff: Change inappropriate #ifdefs (NOICP) around hints to NOHINTS. Fix TYPE to work in K95 with stdout. Fix TRANSLATE and TYPE commands on Little-Endian platforms. Add hint when Telnet negotiations are taking more than 15 sec. SSL changes. cku{con,cns}.c, ckuus[46].c, ckctel.c, ck_ssl.c, 16 May 2000. Changed version number to 7.1.199, since we are now contemplating making the new sources public to testers. ckcmai.c, 16 May 2000. xgnbyte() was originally intended for reading bytes from a file and translating to Kermit's transfer character set. Let's make sure this still works: Platform FCS TCS FCS LE BE latin1 ucs2 latin1 ok ok ucs2 ucs2 latin1 ok ok latin1 latin1 latin1 ok ok ucs2 latin1 latin1 ok ok Why does this work, despite all the recent confusion? Because there is a special test for (what == W_SEND) to use a translation function that returns the bytes in Big Endian order. Added a note to this effect to the xgnbyte() heading. ckcfns.c, 16 May 2000. So when we are not sending a file with Kermit protocol (what != W_SEND), inspection of the code seems to indicate that xgnbyte() returns UCS-2 bytes in native order, i.e. byteorder (not fileorder, not ucsorder). But In typegetline(), we are using fileorder, rather than byteorder, to decide whether to swap bytes before sending them to xpnbyte(). And yet the TYPE code seems to work. Does it really? Using the current code: Arch File BE Latin-1 OK BE UCS2 BE BOM OK BE UCS2 BE No BOM OK BE UCS2 LE BOM OK BE UCS2 LE No BOM OK BE UTF8 OK LE Latin-1 OK LE UCS2 BE BOM OK LE UCS2 BE No BOM OK LE UCS2 LE BOM OK LE UCS2 LE No BOM OK LE UTF8 OK The LE ones were done both in Linux and K95G. So I guess I still don't understand how xgnbyte() works! But it does. Will revisit this as needed. Meanwhile, XLATE didn't work if cset1 == cset2 when Kermit was built with UNICODE defined. Fixed in xlate(), ckuus4.c, 16 May 2000. From Jeff: In IKSD, don't let guests use the ENABLE command. Add code for mouse debugging. Replace lots of strcat()'s with new ckstrncat(). Add K95 command-window status line. Better checking of guest status in zmkdir(). Added SET TELOPT LOGOUT. Add --account: command-line option for IKSD. Fix SET IKS ANON ROOT to parse directory rather than file name. Make default language for SNI 97801 be German rather than North-American. Adjust to new Microsoft NTLM formats. ckuath.c ckctel.c ckcnet.c ckuus7.c ckuus3.c ckuusy.c ckuus5.c ckuusx.c ck_ssl.c ckuusr.c ckufio.c ckuus2.c ckuus4.c ckcmai.c ckuusr.h ckcpro.w ckcdeb.h ckuus6.c ckctel.h, 20 May 2000. There was still one place in the help text where the term "regular expressions" was used (HELP INPUT). These are patterns, not regular expressions (true regular expressions let us match, e.g. zero (or one) or more digits; patterns can't do that). Changed the text to say "help patterns", and added an invisible "help patterns" command. Fixed all other references to "regular expression". ckuus[r2].c, 20 May 2000. CL used to be sufficient to abbreviate CLOSE; fixed it so it is again. Ditto for CL[OSE] P[ACKET-LOG]. ckuusr.c, 20 May 2000. From Jeff: corrections to new K95 command-window status line code, plus some auth stuff. ckcmai.c, ckuus[7x].c, ckuath.c, 21 May 2000. The DELETE command lacked a /RECURSIVE option. Added the switch and runtime actions to dodel(). This allows deleting of all (or selected) files in a tree, but not directories. ckuus6.c, 21 May 2000. Added a /DIRECTORY switch, which means "delete directories too". The objective is to do what DELTREE does in DOS, or "rm -Rf" does in Unix. For this to work, all files must be deleted from each directory before the directory itself can be deleted. I thought this was going to require the construction of an in-memory tree and depth-first traversal of it, and therefore an arbitrary amount of memory, which is why I had been putting it off. But then I realized that given the recursive file list, I can simply sort it in reverse order, which guarantees files will be deleted before their containing directories. This is trivial in Unix, where we have direct access to the "mtchs" array; I don't know about other platforms. Anyway, this works fine in Unix. So: DELETE /RECURSIVE /DIRECTORY /DOTFILES * deletes all the files in the current directory and all the directory trees rooted in the current directory. In K95, of course, the /DOTFILES switch isn't necessary. ckuus6.c, 21 May 2000. Added DELETE /ALL as a shortcut for DELETE /RECURSIVE /DIRECTORY /DOTFILES. ckuusr.h, ckuus[26].c, 21 May 2000. Of course there are minor hitches, like: DELETE /RECURSIVE /DIRECTORY /DOTFILES foo where "foo" is the name of a directory, deletes everything in foo but not foo itself. That's because zxpand(), when given a directory name, treats it as "name/*", so in this case we add the name of the directory to the end of the list. ckuus6.c, 21 May 2000. Next hitch (Unix only): DELETE /ALL xxx, where xxx is a symlink to a directory, follows and expands the symlink instead of deleting it, and therefore deletes the directory tree pointed to by the symlink, not good. That's because when 'recursive' is set, we open directories, and a symlink to a directory passes the isdir() test. DELETE xxx (no /RECURSIVE) deletes the only symlink itself, which is what we want in the recursive case too. To get around this required defining a new ZX_NOLINKS flag bit for nzxpand(), which we must test in traverse(). To pass this option from the user interface to nzxpand() required changing argument 5, 'd', to cmifi2(), to be a bitmask. Bit 0 (still) says whether directories should be included, and the new bit 1, if set, means "don't follow symlinks". Then cmifi() and friends (cmdir(), cmifip()) can form the appropriate flags argument for nzxpand(). ckcdeb.h, ckucmd.c, ckufio.c, ckuus6.c, 21 May 2000. I looked at COPY and RENAME to see what it would take to allow them to accept wildcards. The idea would be that if the destination was a directory name, wildcards in the source would be OK since the destination files could have the same name (I did NOT consider fancier options, such as "rename *.doc *.txt"). But COPY is tough because some of options, like /APPEND, would not fit, and also because of the mixed modes of copying (zcopy() versus inline fopen/fread/fwrite/fclose loops)... RENAME looks more manageable. But in poking at it I noticed a bug: "rename foo bar", where foo is a directory that contains exactly one file, renames that file to bar, rather than the directory itself, oops. This was easily fixed by setting 'd' to 1 in the cmifi() call. ckuus6.c, 21 May 2000. Changed RENAME to allow wildcards in first argument if second argument is a directory name, and then put the zrename() call in a loop, so now RENAME handles multiple files (including directories). Incidentally, nothing would be gained by adding a /RECURSIVE switch, because when you rename a directory, you "move" its contents -- the entire subtree -- to a new location. ckuus[26].c, 21 May 2000. Added checks to zrename() and zcopy() for IKSD with DISABLE DELETE when target file already exists. ckufio.c, 21 May 2000. Discovered that Unix zcopy() could truncate the last character of the destination if it was a directory. Fixed in ckufio.c, 21 May 2000. Discovered that COPY never worked if the target was a directory name and any of /APPEND, /SWAP, /FROMB64, or /TOB64 was given. Fixed in ckufio.c, 21 May 2000. Discovered that COPY, when given the /APPEND, /SWAP, /FROMB64, or /TOB64 options, did not check if the source and destination files were identical; if they are, the original file is clobbered. Fixing this required adding a new API: zcmpfn(s1,s2) - "compare two filenames"; returns 1 if they refer to the same file. ckufio.c, ckuus6.c, ckcplm.txt, 21 May 2000. Discovered that zcopy(), when it discovered the source and destination files were identical because they had the same inode, returned the wrong error code (-3 instead of -5). Fixed in ckufio.c, 21 May 2000. Now back to COPY. Rewrote the whole docopy() routine to allow multiple files if the /APPEND switch is not given and the target is a directory. Reworked all the messages and feedback to be like DELETE. ckuus[26].c, 21 May 2000. From Jeff: Corrections to K95 Command-window status line. ckuusx.c, 22 May 2000. Added code to dodel() for K95 to make a copy of the file list and sort it, since in K95, znext() does not return filenames in any particular order, nor is there a mtchs array that we can sort directly. Also changed the SORT /DIRECTORY switch to /DIRECTORIES, for consistency with the DIR command. ckuus[26].c, 22 May 2000. From Jeff: Changes for HTTP proxies, K95 URL highlighting, Telnet Uservars, Telnet Env Location, SSL things, K95 Command-window status line, ON_LOGOUT macro, fixes to MKDIR permission checking, simplification of WIKSD init file search, disable IKSD user redefinition/deletion/display of ON_EXIT or ON_LOGOUT. ckctel.c ckuus7.c ckuus3.c ckuus5.c ckuus4.c ckufio.c ckcnet.c ckuusx.c ckcmai.c ckuus2.c ckuusr.h ckcnet.h, 25 May 2000. SET COMMAND HEIGHT and WIDTH were broken in non-K95 builds. Fixed in ckuus3.c, 25 May 2000. Fixed "take ?" message not to look wrong in K95. ckuusr.c, 25 May 2000. Fixed expression evaluator to set error indication upon attempt to divide by zero. termp(): ckuus5.c, 25 May 2000. This prevents \feval(\%x/0) from returning a value and makes it give an error, but the error message is misleading ("argument not numeric"). Fixed evalx() to set a special flag whenever divide-by-zero is attempted, and then retooled fneval() to check the flag everywhere that evalx() is called, and return the appropriate message (divide by zero versus arg not numeric). Also caught a couple functions that never did handle evalx() errors right, such as \flpad(). ckuus4.c, 25 May 2000. Also discovered that \fn2hex() and \fn2oct() did not accept expressions, only numbers. Fixed in fneval(), 25 May 2000. From Jeff: Force a prompt in IKSD if negotiated authentication fails; fix resetting of command timer in cmkey(); fix extended cmdline options keywords (xferlog,xferfile were switched) and add "--default-domain" for NT. ckcmai.c, ckucmd.c, 27 May 2000. SEND /COMMAND didn't work because it was checking the command with zchki() for read-accessibility. Fixed that, but it still didn't work because in this case cmarg was not changed to point to the command. Fixed that too. doxsend(): ckuusr.c, 27 May 2000. But now sfile() was having zfnqfp() put a full path on the front of the F-packet name. There was a lot of confusion in sfile() about when the thing it was sending was or was not an actual file. Simplified all the decisions by adding a new variable, notafile, which is nonzero when whenever we SEND /CALIBRATE, /COMMAND, /FILTER, or /ARRAY, and then replacing all the complicated tests with tests of just this variable. ckcfns.c, 27 May 2000. But that's not all. We were also putting bogus size & date info into the A-packet. Oops, this is a tough one. sattr() is called after sfile(). sattr() calls zsattr() to fill in the attributes. But zsattr() ASSUMES that it is only used when sending actual files and that the file has already been opened, its size has already been obtained, etc; the zsattr() API does not include the file name. Boy was that dumb. So all this time, whenever we called zsattr() when sending something other than a file, it was filling the A-packet with garbage. The easy way out is to simply not call zsattr() if it's not a real file. But we WANT certain items filled in, like text/binary mode, etc, and (in the case of calibration) the size (since we know it in advance). Since we can't change the API, we'll have to make sattr() fill in the attribute structure itself if it's not a real file. ckcfn3.c, 27 May 2000. By the way, the bogus data in A-packet problem occurred not only with SEND /COMMAND, etc, but also with *all* server responses to REMOTE commands, since the beginning of time. A day for discoveries! Here's another: Noticed that chkfil() was being called by sfile() on commands, calibration runs, etc. Fixed sfile() to call chkfil() only for real files, and chkfil() itself to return -1 if called during calibration runs or for commands or filters. ckcfns.c, ckuusx.c, 27 May 2000. Dat Nguyen complained about not being able to define macros with names that start with & or %. This goes all the way back to C-Kermit 5A; the \%x and \&x[] variables are kept internally without the \. I thought it might be easy to change this, but it wasn't and I backed off. Way too dangerous. From Jeff: #ifdef's for yesterday's changes because pipesend variable not declared #ifndef PIPESEND. Changed pipesend to be declared always to cut down on the many #ifdefs. ckcfn[s3].c, ckuusx.c, 28 May 2000. At Dat Nguyen's suggestion, added a mechanism for handling unknown commands. If a macro called ON_UNKNOWN_COMMAND is defined, and a top-level command is given that does not match a top-level keyword, token, or macro name, we call cmtxt() to gather operands and then invoke ON_UNKNOWN_COMMAND with \%1 set to the unknown command, and \%2, etc, set to the operands, if any. At Jeff's suggestion, we also take care not to recurse in case the ON_UNKNOWN_COMMAND definition itself contains an unknown command. ckuusr.c, 28 May 2000. Samples: DEF ON_UNKNOWN_COMMAND telnet \%1 ; Treat unknown commands as hostnames DEF ON_UNKNOWN_COMMAND dial \%1 ; Treat unknown commands phone numbers DEF ON_UNKNOWN_COMMAND take \%1 ; Treat unknown commands as filenames DEF ON_UNKNOWN_COMMAND !\%* ; Treat unknown commands as shell commands Note that this trick does not work for strings that are prefixes of existing commands or macros, e.g. "a" is not picked up as an unknown command. Should it be? I think it would be more than a little reckless. In case anybody ever brings it up, though, we can always add ON_AMBIGUOUS_COMMAND. Made ERASE an invisible synonym for DELETE. ckuusr.c, 28 May 2000. Improved "dir ?" message text. ckuus6.c, 28 May 2000. Having DIAL spit out error messages in mid-command if LINE or MODEM TYPE not set was too annoying so I moved them to after cmcfm(). ckuus6.c, 28 May 2000. Added CM_PSH flag definition for keyword flags, meaning "invisible and disabled if nopush/NOPUSH", and CM_LOC for "invisible and disabled if nolocal/NOLOCAL". ckucmd.h, 28 May 2000. Tagged all top-level commands, SET commands, and SHOW commands, SEND and GET switches, with CM_LOC and/or CM_PSH as appropriate. ckuus[r6].c, 28 May 2000. Changed kwdhelp() to not show CM_LOC commands if nolocal is set or NOLOCAL is defined, and not show CM_PSH commands if nopush is set or NOPUSH is defined, ckucmd.c, 28 May 2000. Changed docmd() to check command keyword flags, so even if user types a CM_PSH or CM_LOC command (which is still possible even though it's invisible), it won't be executed. ckuusr.c, 28 May 2000. Thus IKSD, since it sets nolocal and nopush, should now have all connection and system-access oriented commands both invisible and disabled. Changed SHOW FEATURES to show only program name & version ("name, rank, and serial number") if IKSD. ckuus5.c, 28 May 2000. Discovered that a command like this: "remote dir | command" given at the IKSD prompt crashes IKSD. I'm not sure why it crashes, but (a) the command parser is supposed to catch this if nopush is set or NOPUSH is defined, and (b) even if the parser doesn't catch it, zxcmd() should simply fail. The reason the parser didn't catch it was that nopush wasn't being tested in remtxt() (as it was in remcfm()). Fixed in ckuus7.c, 28 May 2000. Added SHOW HISTORY to show the command recall buffer contents. ckuusr.h, ckuus5.c, ckucmd.c, 28 May 2000. Fixed addcmd() (which adds a command to the recall buffer) to check first that last command is not the same -- I assumed it did this, but it never did. Also changed cmhistory() to list commands in normal order rather than reverse order; this way, the user sees the "nearest" command on the bottom, the next nearest above it, etc, in the order that ^P would work. ckucmd.c, 29 May 2000. Added SAVE COMMAND HISTORY [ ]. ckucmd.c, ckuus[r7].c, 29 May 2000. (Check to make sure SAVE { TERM, COMMAND } SCROLLBACK and SAVE KEYMAP still work.) Updated HELP SAVE text. ckuus2.c, 29 May 2000. Added an IKSD version of the top-level help text that does not say inappropriate things, like "Type HELP OPTIONS for help with command-line options", "Type MANUAL to access the K95 manual," etc. ckuus2.c, 29 May 2000. Changed INTRO text to not include hardwired uparrow an downarrow characters in K95 since these don't work in IKSD and also fixed up the formatting and added a warning about using backslashes in PC pathnames. ckuus2.c, 29 May 2000. Worked on TAP/IXO script for AcIS paging: . I had to fix the Multitech init string: %E1 and #L0 not supported by this model. . Something was wrong with the setting of LINE, which I fixed, so now it can dial successfully. . I can get the ID= prompt when I CONNECT and type CR, but the script never gets it. Apparently there is no DSR wire in the cable; no matter whether I set &S0, &S1, or &S2, there is never a DSR signal, and this might have something to do with ttchk() never finding any chars waiting. . But I can dial other numbers, like the CU modem pool, and scripts work just fine, so that's not it either. So maybe it's parity? Aha, the problem was that INPUT calls ttvt() each time, which can be rather disruptive. In this case we have INPUT in some tight loops. Well, ttvt() is supposed to return without doing anything if it's already in ttvt state, but that wasn't working because we were testing too many things. Removing the test for carrier control did the trick -- now the page works. This is probably a 4.2BSD-specific thing, otherwise I'd have have been lynched by now. Fixed in ckutio.c, 29 May 2000. From Jeff: Add missing return() statement to savhistory(), clean up SAVE HISTORY switch statement. ckucmd.c, ckuus7.c, 30 May 2000. Removed %E1 and #L0 from Multitech command strings since they cause errors in the Multitech 224E EC model. ckudia.c, 30 May 2000. Jeff discovered a problem with \fexec(), which can be boiled down to this: If a macro A returns some material concatenated with the return value of another macro B, the leading material is written over by the return value of macro B. Example: define foo return B define bar return A\fexec(foo)C echo \fexec(bar) Prints BBC instead of ABC. Evaluation of \fexec() in bar's RETURN statement causes some stack action, and therefore the return value pointer moves right in the middle of writing the result. Anything to the right of \fexec() is safe, but anything on the left gets overwritten. It turns out that we were using the global line[] buffer and lp pointer to it on the mistaken assumption cmpush()/cmpop(), used by \fexec(), would take care of it. The solution is to use a local automatic buffer (big) and pointer in doreturn() rather than global line[] and lp. I could also have added line[] and lp to cmpush() and cmpop() but that might have had side effects and would have been just as expensive. ckuus6.c, 30 May 2000. Back to TAP/IXO script. Part of the ritual is to send PG1 at the ID= prompt. But if you have INPUT ECHO ON, the P echoes back and, if you have VT220 or above terminal or emulator, it "hangs" because P is DCS. Changed INPUT to use the TERMINAL DEBUG setting to determine how to echo echaracters. Thus if TERM ECHO is ON and INPUT ECHO is ON, the terminal will receive ^[P (three chars) rather than P, and no harm is done. ckuus4.c, 30 May 2000. However, this doesn't work in practice and I'm totally stumped. If I view the session through K95 in debug mode, what I see (this doesn't come across very well in monochrome) is: WAITING FOR ID= PROMPT...MJ M^JNO "ID=" PROMPT (1)[0A] - TRYING AGAINMJ MNO "ID=" PROMPT (1)[0A] - TRYING AGAINMJ M^M^JID=[\RCVD ID=(3)...MJ JENT PG1(1)...M The M's and J's are (red) control chars. At this point, K95 shifts into "I'm in an escape sequence" mode (between the J and ENT), obviously because it received the echo to PG1. But then why doesn't it show it? Meanwhile, note that all the other control characters received by INPUT are correctly echoed as ^J, ^M, ^F, ^B, etc: J10 1.8^M114 Processing - Please Wait^M^F^MRESULT=1M JENT BLOCK (1)...Mn.M JJ^?{u^_+^^SOw^[^?211 Alphanumeric Page 1 Accepted.^M^F^MRESULT=1M J[^D^M+++M JTQ0H0M JTK!@5e\P}]M JO CARRIERM Jlosing /dev/ttyb...OKM A mystery, but before I could get any further the system went down. Anyway, I checked the change on an ordinary Telnet connection, in which I had INPUT echo a stream of all 256 possible bytes, and it worked perfectly. Most of the character-set translation functions that convert from a 7-bit set were failing to check if the source character had its 8th bit set, and in that case would return something incorrect (at best) or make a wild array reference (at worst). Fixed in ckuxla.c and ckcuni.c, 31 May 2000. Added symbols for \fjoin() and FOREACH to ckuusr.h, 31 May 2000. Added \fjoin(&a[,s]), joins the elements of array a together, separated by optional separator s. The array name can include an optional range specifier. If none is given, elements 1 through are joined: decl \&a[] = one two three four five six seven eight nine ten echo \fjoin(&a) all elements, no separator echo \fjoin(&a,:) all elements, separator = ':' echo \fjoin(&a,][) all elements, separator = "][" echo \fjoin(&a[3],:) elements 3 through end, separator = ':' echo \fjoin(&a[3:5],:) elements 3 through 5, separator = ':' echo \fjoin(&a[3:5],{,}) elements 3 through 5, separator = comma. echo \fjoin(&a[3:5],...) elements 3 through 5, separator = "...". ckuus[24].c, 31 May 2000. From Jeff: fix a typo in HELP FUNC JOIN, minor fix to Pragma Systems Telnet client, Add --syslog: cmdline option for IKSD. ckctel.h, ckuus[2y].c, 1 Jun 2000. Here is a really interesting problem. "send /command { xxx }". If "xxx" contains an equal sign or colon, it is doubled. Why? Because we are still in cmfdb() and are trying to parse a switch; therefore "=" and ":" are break characters and gtword has to double break characters for reasons that are too hard to explain. In this case, gtword() has been called by cmswi() with brk=1 so what else can it do? It can be a little smarter about when to obey the brk argument. If the first character in the field that it's parsing is not a slash (/), then the field can't possibly be a switch, and therefore the brk arg can be ignored (note that this determination is made *before* stripping braces). ckucmd.c, 1 Jun 2000. Spaces around function arguments should be ignored, but they were not being ignored in all cases, e.g. in "echo \freplace(abc, {b}, x)". Fixed in fneval(): ckuus4.c, 1 Jun 2000. Added \fsubstitute(s1,s2,s3), which is just like Unix 'tr'. s1 is the source string; s2 is a list of characters to be translated, s3 is the list of characters to translate them to; s2 and s3 can be (or contain) ASCII ranges, like \fsubstitute(\%a,[a-z][A-Z]). If s2 is shorter than s3 (after expansion of ranges), then any characters in s2 that don't have corresponding characters in s3 are removed from the result. If s3 is longer than s2, the excess characters are ignored. Examples: echo \fsubstitute(abcdefg,c,x) ; Changes "c" to "x" echo \fsubstitute(abcdefg,cd,xy) ; c -> x, d -> y. echo \fsubstitute(abcdefg,[c-e],[C-E]) ; Uppercase c thru e echo \fsubstitute(abcdefg,a[c-e]g,A[C-E]G) ; Uppercase acdeg echo \fsubstitute(abcdefg,c,) ; Removes "c" define \%a abcdefghijklmnopqrstuvwxyz echo \fsubstitute(\%a,[f-h][p-r],[F-H][P-R]) ; Multiple ranges OK echo \fsubstitute(\%a,[p-z]) ; Can delete a range assign dosline This is a DOS line\13\10 ; Handles control chars echo \fsubstitute(\m(dosline),\10\13,JM) .\%w = a ; Handles variables .\%x = z .\%y = A .\%z = Z echo \%a echo \fsubstitute(\%a,[\%w-\%x],[\%y-\%z]) Can also be used with 8-bit characters too, so it's quite possible to use this function to translate single-byte character sets if you don't mind setting up the s2 and s3 strings yourself. ckuusr.h, ckuus[24].c, 1 Jun 2000. From Jeff: minor tweaks to ckuus[5x].c, plus a new linux+pam+shadow makefile target. 4 Jun 2000. Cleaned up some more confusion with command recall and SHOW HISTORY. ckucmd.c, 4 Jun 2000. Fixed \v(m_aa_off), which was erroneously returning the value of \v(m_ec_off). ckuus4.c, 4 Jun 2000. Refer to notes of 24 Sep 1999, in which we introduce the "packet zero hack" to alleviate the problem with overloading of Packet 0 in the client/server setting, e.g. client sends I packet, server sends ACK(I), client sends G Packet (also number 0) but client resents ACK(I). How does the server tell the difference between ACK(I) and ACK(G)? To do this, we saved a copy of the data field of ACK(I), and then when we get an ACK to the next packet (such as G) we compare its data field with ACK(I)'s data field, and if they are identical, we know that ACK(I) was retransmitted, so we send our G packet again. There was at least one flaw in this reasoning: what if the server sends an empty ACK(I), which is perfectly legal? An empty ACK(I) is identical to ACK(G), so we can't use the trick in this case (if we do, the client will keep sending packets to the server after the server has quit from packet mode; this occurs with KEA!VT340). Fixed in ckcpro.w, 4 Jun 2000. Added an optional 4th argument to \freplace(): a number that specifies which occurrence of the target string to replace. If 0 (the default), all occurrences are replaced. If 1, the 1st occurrence is replaced. If 2, the 2nd occurrence is replaced. And so on. If less than 0, occurrences are counted from the right. ckuus[24].c, 4 Jun 2000. WIKSD code shuffling from Jeff. ckcmai.c, 5 Jun 2000. Fixed various syntax problems, ifdefs, and missing prototypes for DECC on VMS. ckcdeb.h, ckclib.h, ckcnet.c, ckcfns.c, ckuus[24].c, 5 Jun 2000. The most baffling of these was DECC insisting that the zstr, zattr, and filinfo structs were being redefined by ckcdeb.h (but only when included from the ckv*.c modules) even though they were not. This is not a conflict with some VMS header file, because I'm building on the same VMS system with the same header files as before. Moving these struct definitions up higher in ckcdeb.h cured the problem. It's not science. Ditto for no-TCP/IP builds, in which some non-TCP/IP material (like doend()) was erroneously included in #ifdef TCPSOCKET. ckuus[r45y].c, ckcnet.c, 5 Jun 2000. Should C-Kermit, when sending a file group and encountering a file that it can't open for reading, fail and stop or just go on to the next file? I thought it should fail and stop (which it does), so as not to mislead the user into believing that all matching files were sent, but Jeff argues there should at least be an option to proceed to the next file when this happens. I took a look at the code to see what would be involved, and found that the code was already written to do this. In gnfile() (get next file from our internal list of files to send), we call zchki() on the file, and if it says the file is not readable, we simply go on to the next one. So the trick is to make sure that zchki() always returns the appropriate code (-3) when the file can't be opened for reading, e.g. when the file is "locked". From Jeff: fix array refs in debug() statements in addcmd(): ckucmd.c; adjust #ifdefs in dotnarg(): ckuusy.c, 6 Jun 2000. From Jeff: "More work on telnet authentication. On Unix we can assume that if an administrator compiles C-Kermit to support an authentication method that the administrator will configure that authentication method so that it can successfully be used. (although if C-Kermit is prebuilt by Linux vendors this may not be true.) On Windows, since we distribute support for all authentication types this is never true. Therefore, there needs to be a runtime test to determine if the authentication method has been configured for server side use. The tests installed are rough but they are better than nothing." ckuus2.c, ckuath.c, 6 Jun 2000. Looking again at the sending-unreadable files problem. Suppose we send the server a command "get blah", where blah is the name of a file that we can list but not read. The server says "File not found" instead of "Read permission denied". When the server gets the filename (in sgetini()), it simply stores it for later retrieval by gnfile(). So later, when sinit() calls gnfile(), gnfile() looks at its list and gets "blah" from it, does all the GET-PATH and zfnqfp() processing, and then calls zchki() on the result. zchki() correctly returns -3 ("read permission denied"). But that doesn't make gnfile() return this error. Instead, gnfile() continues its loop through the file list, and finds there are no more filenames in it, so returns 0 (it might just as easily have found more files, or switched to a different list). But if gnfile() returns 0 to sfile(), then we know we have to send an error packet, so the trick is to make gnfile() set a global status variable whenever it gets an error, and then later, if it returns 0, sfile() can check the status variable to find out the real error. Other callers can (and should) ignore gnferror, however, because once we have started sending files, we skip any that we can't open. ckcfns.c, 6 Jun 2000. In a similar vein, when C-Kermit is in remote mode and you give it a SEND command for a file or group that does not cause a parse or zchki() error, but for which gnfile() returns 0, it printed no sensible error message, and then went on print an obnoxious and misleading hint. Added another clause to sfile() to catch this case. Example: "send /except:x* x*". ckcfns.c, 6 Jun 2000. Building on the VAX with VAXC, I got a whole different set of duplicate declarations (typedefs this time) in ckcdeb.h when included from ckvfio.c: MACRO, KEY, and WAIT_T. Now I see why (and why I had the trouble with the struct redefinitions yesterday): ckvvms.h contained an #include for ckcdeb.h, which is entirely unnecessary, since every module already includes ckcdeb.h. Now, what's funny about this is that ckcdeb.h protects itself against multiple inclusion, and yet these typedefs and struct declarations are being re-executed anyway. What a stupid language. ckvvms.h, 6 Jun 2000. Back to the trying-to-send-an-unreadable-file problem... What if the original filespec passes the gnfile() in sinit(), so we have a list of files to send, and then later gnfile() hits a file that zchki() passes (and so gnfile() returns it), but sfile() still fails to open it? This is evidently happening in Windows when a file is "locked" (e.g. open, mapped, or whatever, by another process), and since some files in Windows are always in this state, any attempt to (say) back up your C: disk with "send /recursive c:/*.*" will always fail as soon as it hits one of these files. As noted previously, the real solution is to fix zchki() to fail if open() would fail. But presently, the code pretty much assumes that if gnfile() returns a filename, the file can be sent, and so recovery from failure to open it is pretty crude. I replaced the code in Y{} in the protocol module with a loop that keeps getting the next file and trying to open it until (a) a file is successfully opened, in which case it is sent; or (b) there are no more files, in which case we move on to EOT state. This also required a small change in sfile(), to not call nxtpkt() (which increments the packet number and sets up the buffers) until and unless the openi() call succeeds. The old code remains in #ifdef COMMENT. Needs testing, especially on Windows with locked files. ckcpro.w, ckcfns.c, 6 Jun 2000. Note: FTP skips over unsendable files too. HELP SET IKS text from Jeff. ckuus2.c, 7 Jun 2000. Jeff noticed that chkfil() was diagnosing Linux i386 binaries as UCS-2. Added another criterion to chkfil(): if a file contains a run of 3 or more NULs, it can't possibly be text of any kind. A UCS-2 text file, however, can easily contain 2 NULs in a row (e.g. U+2500 U+0041). This test takes precedence over the BOM. Also, if the file starts with a UTF-8 BOM, the BOM is not believed if the file contains any NULs at all, since text files don't have NULs. This change also handles the case where a non-Unicode file happens to start with the UCS-2 or UTF-8 BOM bytes -- wouldn't it be funny if these showed up somewhere in (e.g.) /etc/magic? Also, added debug() statements to chkfil() to indicate reason for result. ckuusx.c, 7 Jun 2000. Changed DIR /XFERMODE to show the result of chkfil() if FILE INSPECTION is ON. This gives a very easy way to test chkfil() on a huge number of files. ckuus6.c, 7 Jun 2000. This revealed two more problems with chkfil(). First, certain hex files were diagnosed as UCS-2 because of the alternating bytes rule. But hex files do not have NULs or other unusual C0 controls, so before deciding a BOMless file was UCS-2, I added the requirement for unusual C0 controls. Second, JIS-7 files were being diagnosed as binary because they contained non-textual C0 controls (Esc, SO, and SI). Added these three chars to the list of C0 chars OK in text files. ckuusx.c, 7 Jun 2000. For docs: Here is some reasons why you might want to SET FILE INSPECTION OFF: . A file contains some other kind of file, e.g. an email message containing a Kermit packet log of a UCS-2 file transfer. . A Postscript file (text) has an embedded graphic (binary). From Jeff: adjust NOLOCAL vs DNS_SRV ifdefs in ckcnet.h; change "set term print transparent" to "set term print user" in ckuus[27].c, initialize file-list elements to NULL in dodel() ckuus6.c, eliminate some cutesy capitalization in HELP OPTIONS in ckuusy.c, new AIX makefile targets + SSL corrections in some others; auth stuff in ckctel.c and ckcnet.c, 11 Jun 2000. Added transaction log entries for files transferred with external protocols. The external protocol is responsible for the group, so we can only log the command and result, but we can't log anything on a per-file basis. ckcpro.w, 11 Jun 2000. Updated character-set associations. FCS UTF8 or UCS2 gets TCS UTF8. FCS KOI8U or KOI8R gets TCS Latin/Cyrillic. ckuxla.c, 11 Jun 2000. Changed SET FILE INSPECTION to SET FILE SCAN. Added an optional size parameter after ON, e.g. SET FILE SCAN ON 8192. The default is 4096, as before. If a size of -1 is given, the entire file is read. A size of 0 makes little sense, but is accepted. Changed chkfil() to accept the scan length as a third parameter. ckcker.h, ckuus[r2467x].c, 11 Jun 2000. Jeff noticed that ckcdeb.h blew up if you tried to build with -DNODEBUG. There was a missing #endif at the end of the IFDEBUG section, but supplying it threw the rest of the file out of whack. A rather prolonged search revealed the following fragment at line 1723: _PROTOTYP(int dodebug,(int, char *, char *, long)); _PROTOTYP(VOID dohexdump,(CHAR *, CHAR *, int)); #endif /* DEBUG */ These three lines just simply did not belong there. They had been "moved" to around line 4182, but somehow this fragment was left behind. So what does the #endif match? The #ifndef CKCDEB_H directive at the beginning. So this explains the typedef and struct errors I got in VMS recently. Luckily it should have no other affect since this fragment was located at "top level". And yes, it has been sitting there for some time, at least since 7.0.197. ckcdeb.h, 12 Jun 2000. Renamed chkfil() to scanfile(). Moved filename-pattern code to matchname(), and changed scanfile() to call matchname itself, so that all decisions about which methods to use and how to use them are in one place. Increased default file-scan size to 48K (from 4K). ckcker.h, ckuusx.c, 12 Jun 2000. Now, what about the relationship of scanfile() and matchname()? scanfile() always comes to a conclusion, so there's no point in considering patterns if FILE SCAN is ON. If scanfile() is wrong, you can either increase the size of the scan or disable scanning altogether and use patterns (or force an explicit transfer mode). I can't think of any reason to do both, rather than one or the other; even if the two methods get different results, which would you choose? File scan, of course, because it's based on detailed information specific to each file. Therefore, changed scanfile() to call matchname() only if FILE SCAN is OFF. This required adding another file type, FT_TEXT, meaning "some unspecified kind of text". ckcker.h, ckuusx.c, 12 Jun 2000. Moved all pattern-related code and definitions from ckcmai.c to ckuusx.c so everything relating to patterns is one place. 12 Jun 2000. Simplified sfile(), DIR /XFERMODE, and ADD SEND-LIST to call only scanfile(). ckcfns.c, ckuus[r6].c, 12 Jun 2000. Changed matchname() to handle filenames with backup suffixes correctly, which it never did. So now if "*.txt" is a text pattern, "foo.txt.~1~" is properly recognized as text. ckuusx.c, 12 Jun 2000. Updated HELP SET FILE text again. ckuus2.c, 12 Jun 2000. Added a rudimentary GREP command. Just a little bit of parsing in ckuusr.c, but cc on watsun dies with "virtual memory exhausted". False alarm maybe; moved the parsing code to ckuus6.c, and after a while it stopped happening but maybe it would have anyway. Anyway, for the first cut, it's simply "grep ". The is a ckmatch() pattern, except '*' is implied at the beginning unless it starts with '^' and also at the end unless it ends with '$'. Thus "grep ^/ *.c" lists all lines that start with slash, and "grep \;$ *.c" lists the lines that end with semicolon (which must be quoted, otherwise it starts a comment). If the pattern contains spaces it must be enclosed in braces: "grep {this is a pattern} *.txt". ckuusr.[ch], ckuus6.c, 12 Jun 2000. From Jeff, 13 Jun 2000: . Fix NOXFER builds: ckcnet.h, ckuus[r567x].c, ckcfns.c. . Fix NOGREEK builds: ckuxla.c. Checked NOCYRIL, NOKANJI, NOHEBREW, NOLATIN2, all OK. NOUNICODE needed a few fixes: ckuus[6x].c, 13 Jun 2000. Added FIND and SEARCH as synonyms for GREP. GREP is visible in UNIX and OS-9; FIND is visible in elsewhere, SEARCH is invisible but included for comfort to VMS users (although in VMS the args are backwards). Made ASKQ invisible so we don't overflow the basic 24x80 screen with ? at top level. Merged ASK and ASKQ help text and removed a no-longer-true sentence from it. ckuus[r2].c, 13 Jun 2000. Added GREP switches /COUNT, /DOTFILES, /NAMEONLY, /NOBACKUP, /NOCASE, /NODOTFILES, /NOLIST, /NOMATCH, /NOPAGE, /LINENUMBERS, /PAGE, and /RECURSIVE switches, so now it does pretty much whatever Unix grep does, plus it can recurse and skip backup files. Also added HELP GREP text. ckuus[26].c, 13 Jun 2000. In testing the GREP command, I noticed a bizarre phenomenon with askmore(): at some point it begins to read characters from who-knows-where. I discovered this with "grep { $} *.txt" in a huge directory -- can't seem to reproduce it any other way (if I change the pattern OR the filespec, no problem). To make a long story short, askmore() calls cmdgetc() which (in this case) calls plain old C-Library getchar(), and getchar() returns the characters of one of the filenames, which obviously have been stuffed into the stdin buffer somehow. Now this is pretty strange, because not only would Kermit have to overwrite the buffer with a filename (or overwrite the buffer pointer), it would also have to set the _cnt variable to the exact length of the filename, which is more than a little farfetched. What seems more likely is that the filename is being somehow echoed back to C-Kermit from K95, but the same thing happens if I put K95 in debug mode (this is starting to remind me of the unresolved problem from May 30 with the pager script putting K95 into DCS mode even when the emulator is in debug mode), and anyway, I don't see any funny characters coming to K95 from the host (like Ctrl-E). Could it be a Telnet ECHO glitch? I can't catch that happening either. Another unsolved mystery. Removed temporary debug() statements from ckucmd.c, ckuus[4x].c, 13 Jun 2000. From Jeff: More NOXFER fixes; a Forward-X tweak; move misplaced #endif so TCP port can be specified in SET HOST / TELNET commands in NODIAL builds; don't give Telnet negotiation hint if IKSD; #ifdef out a lot of code if built with IKSDONLY (X.25, Rlogin, Pipes, NetBIOS, PTYs, LAT, Browser, HTTP, etc); fix a lot of #else and #endif comments. ckcdeb.h, ckcnet.h, ckctel.c, ckuus7.c, ckuath.c, 14 Jun 2000. After these changes, yesterday's problem vanishes. If (when) it comes back, I'll put some more effort into tracking it down. A newsgroup posting this morning said "xlate foo*bar" didn't work if the "*" was actually part of the filename. Of course not, you have to quote the asterisk. But that doesn't work either (it used to in 6.0). Another day, another puzzle. Well, I'm not sure how it was broken, but I'm also not sure how it ever could have worked. The fix for this has to be at a very low level, in the dreaded traverse(), to preserve the distinction between quoted and nonquoted metacharacters in case they are mixed in the same filespec (something which never worked). Upon entry, traverse() checks to see if the filespec is wild by calling iswild(), which in turn must (and does) check for quoted metacharacters, so does not return a false positive for (e.g.) "foo\*bar". So far so good. Now traverse() knows it has a nonwild filename and so can return a one-element list containing only this name, rather than opening and scanning the directory for matches. BUT... Instead of simply copying the name literally to the list, it must strip out backslashes. Of course now we get into trouble with files that have backslashes in their names, but at least you can work around this by just typing more of them. Examples: C-Kermit>dir foo* -rw-rw---- 2246 2000-06-14 17:06:43 foo*bar -rw-rw---- 2246 2000-06-14 17:06:47 foo*baz -rw-rw---- 2246 2000-06-14 17:13:09 foo\bar -rw-rw---- 2246 2000-06-14 17:13:11 foo\baz C-Kermit>dir *\** -rw-rw---- 2246 2000-06-14 17:06:43 foo*bar -rw-rw---- 2246 2000-06-14 17:06:47 foo*baz C-Kermit>dir *\\\\* -rw-rw---- 2246 2000-06-14 17:13:09 foo\bar -rw-rw---- 2246 2000-06-14 17:13:11 foo\baz C-Kermit> traverse(): ckufio.c, 14 Jun 2000. From Jeff: make \v(ip) be an acceptable short form for \v(ipaddress); some authorization stuff; some SHOW FEATURES corrections/additions; ckuus[r457].c, ckcdeb.h, 15 Jun 2000. Several years ago I spent a few minutes trying to enable doublequote enclosure of filenames and other fields that might contain spaces, but gave up after a series of entanglements. Let's give it another shot. First, in gtword() we set a flag if the first nonblank character of the field is '"'. When the flag is set, we don't break on space unless the preceding character was also an unquoted '"' (and not the first one either!). Then I changed brstrip() to strip not only enclosing {}'s but also enclosing doublequotes. And then I changed setatm() to not break on space if the string starts with a doublequote unless the space follows another (not the same) (unquoted) doublequote. This pretty much works: . dir "this file" . cd "this dir" . if equal \%a "a b c" echo blah and so on. The various demo and torture-test scripts still work. However: . echo "foo" now echoes foo without the quotes, which is an incompatible change from all previous versions. . echo \"foo\" also strips quotes, which is obviously not right. Note that doublequotes are not equivalent to matched braces: . They have effect only at outer level; there is no concept of nesting. . They don't work for MINPUT args or macro args. Also some open questions: . Should fields enclosed in quotes be immune from \-evaluation? . Should filenames enclosed in quotes be immune from wildcard expansion? I'll have to do a lot of testing, debugging, and maybe changing in the next few days. ckucmd.c, ckclib.c, 15 Jun 2000. From Jeff: a minor correction to auth parsing. ckuusr.c, 16 Jun 2000. Back to quoting... Why does 'echo \"foo\"' not print the doublequotes? For the same reason that 'echo \{foo\}' doesn't print its braces. ECHO sends its argument to zzstring() for evaluation, in which \" becomes ", so by the time brstrip() gets its hands on it, the backslashes are gone. Can we live with this? Sure, since the same thing happened before with braces. Solution: put an extra set of doublequotes (or braces) around the text. Next, is it OK to introduce this incompatible change? It affects not just ECHO but any other command that takes a text string (cmtxt()) as an operand, such as WRITE, FWRITE, etc. The conservative answer is no, existing behavior must be preserved. But how many times do people really want to echo (write, etc) a string enclosed in doublequotes? More to the point, if we make an exception for ECHO, WRITE, etc, then our quoting rules become inconsistent, whereas now you can pretty much say that any text string can be enclosed in doublequotes to force it to be considered as a single field, which is intuitive to users of almost any shell -- DOS, UNIX, etc. Let's take a census of where brstrip() is used to doublecheck... . Modem commands and dial strings. . SET KEY and SET TERMINAL KEY definitions. . SET PROTOCOL fields . SET PRINTER device and end-of-job string . SET TELNET PROMPT (for user ID) . SET PROMPT . SET various authentication prompts, login ID & password, etc. . DEFINE variable definition . GREP string and filename <-- Doesn't cmifi() already strip? . SEND/GET /RENAME:string /MOVE-TO:string /AS-NAME:string /FILTER:string . SEND/GET/RECEIVE filenames and as-names . SEND and RECEIVE filters . IF fields . SORT ranges . TEXT and BINARY PATTERNs . REMOTE blah redirection or pipe strings. . SET ROOT string . TRIGGER strings . IDLE-SEND string . ECHO string . RETURN value . EXIT, STOP, and END messages . OUTPUT string . INPUT and MINPUT . EXEC command and args . APC text . TYPE /PREFIX:text . WRITE and FWRITE text . MKDIR, RMDIR directory name None of these strike me as particularly dangerous places for doublequoting, and most of them are places where people would expect to be able to do it. The reason quotes didn't work on MINPUT strings is that the MINPUT parser wasn't calling brstrip(). Replaced clunky code with a call to brstrip(), now everying is fine. ckuus4.c, 16 Jun 2000. Let's check to see what else isn't calling brstrip(). A search for "'{'" in ckuusr.* turns up about 50 hits: . SET DIAL DIAL-COMMAND: Fixed in ckuus3.c. . Function argument parser -- not sure if I should touch this. . Macro block parser -- don't touch. . xwords() -- this makes the macro argument list: ckuus5.c -- deferred. . The DIAL number: No, because of {{xxx}{yyy}{zzz}} notation... That's about it. So now the questions are: . Allow doublequoting of function args? No, too dangerous for lots of reasons. It would only muddy the waters as to precedence of braces, quotes, commas, parens, etc, plus braces and parens match up naturally left and right, which is important in a context where nesting is common. . Allow doublequoting of macro args? This would seem to be less dangerous, worth a try... But first, it seems that xwords() has some problems even as it stands. It makes no allowance for quoting of braces, unbalanced braces, etc. Therefore it is impossible to send (e.g.) a literal left brace as an argument to a macro; nothing works: \{, \123, etc. Doublequotes would have the same problem. Maybe the way out of this dilemma is to say that if a field begins with a doublequote, we ignore interior braces, and vice versa. OK, let's try that... Hmmm, yes, it works very nicely: "{" and {"} are both accepted as single-char macro arguments. And now you can also have arguments with unbalanced braces like "{{{{", odd numbers of doublequotes like {a"b"c"d} But we also get some unexpected effects, like: define xx show args xx a b ""c d e"" f The args are passed as a, b, NULL, c, d, e"", and f. But we had the same problem before with: xx a b {}c d e{} f The problem is that if a field begins with { or ", it terminates on the matching } or " even if it is not followed by space or EOL. OK, that's easy to fix, at least for the quotes. As for braces, what would we reasonably expect the arguments to be in cases like these: xx a b {}c d e{} f xx a {b}c{d} e The only sensible thing is to keep it simple and consistent. If a field begins with "{" we always strip it and collect characters (ignoring spaces) until we reach the matching "}". But if the matching right brace is not the end of the field, as in the cases above, we can't delete it -- the field is not finished yet. Therefore the second argument in the first case above is "}c" and the 4th arg is "e{" The final right brace is removed because it matches the first right brace. In the second example, the second argument is "b}c{d" as expected. Other effects: {a b c} is one argument, a b c "a b c" is one argument, a b c {{a b c}} is one argument, {a b c} "{a b c}" is one argument, {a b c} ""a b c"" is one argument, "a b c" {"a b c"} is one argument, "a b c" This all seems pretty reasonable but needs a lot of testing, but so far so good -- the demos and torture tests still work. xwords(): ckuus5.c, 16 Jun 2000. Changed the strategy for braces to allow a closing brace to terminate a field only if it is followed by whitespace or occurs at the end. So now: xx {}a b c{} has one argument: "}a b c{". Suggested by Jeff. This is more consistent with doublequotes anyway, since a doublequote doesn't terminate a field unless it is followed by whitespace or is at the end. ckuus5.c, 17 Jun 2000. Added a compile-time symbol to disable doublequotes: NODOUBLEQUOTING. ckuusr.h, ckuus5.c, 17 Jun 2000. Added runtime command to disable/enable doublequoting in case it interferes with any existing scripts: SET COMMAND DOUBLEQUOTING, and added it to SHOW COMMAND. ckuusr.h, ckuus[235].c, 17 Jun 2000. After xwords() changes (for macro arguments). checked everything else that calls xwords(): K_* environment variable parsing (should be fine), SHOW CONNECTION (is fine), old/new-format dialing directory conversion (fine), dialing- and network-directory lookup (...), and in the KERMIT command (where allowing doublequotes is an improvement). The LOOKUP and DIAL commands now can find dialing-directory entries whose names contain spaces and are enclosed in doublequotes, but you can't use doublequotes in the DIAL or LOOKUP commands themselves. This was fixed in dodial(): ckuus6.c, 17 June 2000. As to the question of whether doublequotes should suppress wildcard expansion of the interior string... No, too much magic and overloading. Doublequotes, like braces, are to group items together that normally would be separate; backslashes (and \fliteral(), etc) are to suppress/control evaluation of the string itself. Although one user raised an interesting point: he assumed that \fliteral() would suppress wildcard expansion, which would seem to make sense. This one would be kind of tough to implement, so let's defer it. Added code to fix broken CRTSCTS definition in BSDI, enabled via -DFIXCRTSCTS (added to BSDI makefile entries). Here's what's in BSDI : #define CCTS_OFLOW 0x00010000 /* CTS flow control of output */ #define CRTSCTS CCTS_OFLOW /* ??? */ #define CRTS_IFLOW 0x00020000 /* RTS flow control of input */ CRTSCTS should be defined as (CCTS_OFLOW|CRTS_IFLOW). Symptom is massive data loss in the incoming direction. Reported by Steven Schultz. I checked FreeBSD 4.1 (also derived from 4.4BSD) and it has the right definition. So does OpenBSD 2.5. NetBSD 1.4 uses a different scheme, no change needed; ditto for Linux. Built OK on BSDI, FreeBSD, and OpenBSD but no way to test. ckutio.c, makefile, 17 Jun 2000. Added FreeBSD 4.1 makefile target to see if 4.1 fixes the setbuf()/ncurses foulup. It doesn't; the -DNONOSETBUF flag is still needed. makefile, 17 Jun 2000. In UNIX, file-open error messages printed by perror() were often missing the filename. This was caused by a coding error in zopeni() introduced with the syslogging feature. Fixed in ckufio.c, 18 Jun 2000. There's no reason why TRANSLATE (XLATE) should not work on multiple files. Added code to allow this, but got "virtual memory exhausted" again in ckuusr.c in SunOS. Evidently docmd() is right on the edge. In the 7.0 build cycle we got a lot of optimizer warnings about it. For now, just moved all the XLATE parsing code to a new routine, doxlate(), in ckuus4.c, and updated HELP XLATE text in ckuus2.c. 18 Jun 2000. Back to docmd(). Moved inline code for each command that has a lot of it out of docmd() to separate static top-level routines in the same module: doclear(), doeval(), dotelopt(), doedit(), dobrowse(), doredo(), domanual(), doassoc(), dohttp(), dotrace(), doprompt(). ckuusr.c, 18 Jun 2000. The filescan() routine opens up an interesting possibility: the ability to SEND only binary files (or only text files). There's no good way to do this with wildards, patterns, etc (since filetypes like ".doc", ".hlp", ".ini", and ".com" are notoriously ambiguous). Sending only binary files or only text files is desirable for several reasons: the receiver might not support A-packets; you might want to send text and binary (e.g. source and object) files to separate directories, etc. Let's try it... . Added SND_TYP definition to ckuusr.h. . Added /TYPE: switch to SEND and MSEND option tables. . Moved fileselect() from ckclib.c to ckuusx.c since it depends too much on other Kermit stuff to be a ckclib routine. . Added code to fileselect() to make the selection. . Updated HELP [M]SEND text. Seems to work OK. ckuusr.h, ckcker.h, ckclib.h, ckuus[rx].c, 18 Jun 2000. Added /TYPE: switch to DIRECTORY. ckuusr.h, ckuus[26].c, 18 Jun 2000. Now we can also add /TYPE: switches to DELETE, COPY, PURGE, etc -- any command that accepts file-selection switches (I'll do that later). Removed some per-character debug() statements from xgnbyte(). ckcfns.c, 18 Jun 2000. Unicode pasting fixes from Jeff. ckuus4.c, 19 Jun 2000. Corrected a bad typo in ckcmai.c, in which a semicolon after "int xfiletype" terminated a declaration list prematurely (this is just a couple days old). 19 Jun 2000. Some syntax fixing: . Supplied some missing commas to HELP text string arrays. . Shuffled some #ifdefs to fix NOPATTERNS and no-TCP/IP builds. . Fixed dogrep() not to try initializing its keyword structure -- that's an ANSI-only feature; moved the keyword table to top level. Built OK on HP-UX and VAX/VMS 5.5 (non-ANSI compilers). Still get optimizer warnings about docmd() from VAXC 3.2, oh well. ckuus[r26].c, ckcpro.w, 19 Jun 2000. Adapted scanfile() to VMS, crudely. In this case we have to call zopeni() and zclos() on each file. zopeni() contains the code to get the file's record format, and sets the global 'binary' variable to 1 if the record format is fixed or undefined (or if FILE TYPE is currently IMAGE or LABELED), otherwise to 0. If the file is binary, we skip the scan. After testing this, though, it seems we must always skip the scan, since text files are likely to be stored in records with binary headers. Anyway, now at least DIR /XFERMODE tells the truth (sort of), and SEND /TYPE: works. The main problem is that object files (*.OBJ;*) seem to be text files because they have variable-length records. But this has always been the case, and while scanning would properly detect .OBJ files as binary, we still couldn't send them correctly since the record boundaries would be lost, plus we'd also get a lot of false binaries on text files. ckuusx.c, 19 Jun 2000. From Jeff: . Rearrange DEBUG/NODEBUG/IFDEBUG #ifdefs in ckcdeb.h. . Move new declarations outside #ifndef NOXFER clause in ckuusx.c. . Rearrange some switch/case material in dotelopt(): ckuusr.c. 20 Jun 2000. A user reported that transparent print (XPRINT) didn't work in Linux. Due to a list-minute "optimization", the printer (fork) was never closed. OK, easy to fix, move one statement, printing works again. But there's more. On a K95-to-Unix-to-remote connection, if I initiate transparent printing from the remote (causing the file to be printed correctly in Unix), K95 hangs. Recall that C-Kermit passes the printer-on and printer-off escape sequences through to the screen (to avoid deadlocks, infinite waits, etc), so the terminal sees a printer-on-printer-off sequence with no data to print. K95 hangs because these two sequences are coming out in reverse order. Why? Because our special case for outputting the printer-off sequence (instead of sending it to the printer) called conxo() (a front end for write()) directly, rather than going through the Unix CONNECT module's internal buffering mechanism. Fixed in ckucns.c (we don't support XPRINT in ckucon.c or in VMS, etc). Also added XPRINT to SHOW FEATURES list. ckucns.c, ckuus5.c, 20 Jun 2000. From Jeff: adjustment to telnet negotiation timeout hint. ckctel.c 23 Jun 2000. dncnv() had a memory leak due use of sprintf() -- anybody could give a phone number or dialing directory entry name that was longer than about 200 and crash C-Kermit. Fixed in ckuus6.c, 23 Jun 2000. There's another memory leak here: set host <1500 A's> but the location isn't obvious. Will look some more later. Momentary panic when testing the safeguard against setuid root operation: it didn't work on SunOS. But that's normal -- SunOS has the old 4.2BSD s[ug]id facilities. It's fine on modern OS's. A Linux Bugtraq report highlighted the number of sprintf's in C-Kermit. Every single one of these that includes a "%s" is a potential buffer exploit. This can be avoided by using snprintf(), but snprintf() is not widely available, and some of the available versions don't work (e.g. the size argument is ignored). Several people pointed out the existence of the plp_snprintf() package by Patrick Powell. I took a look at it, and concluded it would be a full-time job getting it to build everywhere that C-Kermit builds. It will be much easier to redo our own sprintf's one by one, by hand, even though there are 881 of them. OK, here goes. The general strategy is: a. If the data to be sprintf'd is only constant strings and/or numbers, verify that buffer is big enough to accommodate the longest possible result. b. If the sprintf is already length-checked (as many are), check the check and fix if necessary; otherwise: c. If the data includes string pointers or buffers, check that no possible value of a string could cause buffer overflow; e.g. if the string is a file specification that comes from the OS (e.g. from realpath()) it can't be longer than MAXPATHLEN. If there is any doubt, recode. ckcfns.c (55): Many are numeric only, so OK. Those involving strings were checked. Most of them were sprintfs to funcbuf[], used in server responses to REMOTE blah commands. These usually consisted of fixed and/or numeric parts (no problem) and variable string parts, which were usually a pathname. However, in all cases funcbuf[] is big enough to hold the longest possible result, e.g. a directory listing line with a maximum-size file/pathname, because that's what it was designed for. In some other cases we are constructing arbitrary error message strings for E-packets. For such cases I made a new C-Kermit library routine, ckmakmsg(), which writes up to four strings into a buffer with length checking, and replaced a number of unguarded sprintf's with ckmakmsg() calls. ckcfn2.c (5): Numeric only, OK. ckcfn3.c (21): All sprintf's not verified safe were replaced with ckmakmsg(). ckclib.c (3): Numeric only, OK. ckcmai.c (2): Version strings only, OK. ckcpro.w (5): All of these needed attention, OK now. ckcuni.c (1): Numeric, OK. ckucmd.c (11): Replaced all string-copying ones with ckmakmsg(). ckucns.c (14): Replaced all string-copying ones with ckmakmsg(). ckucon.c (13): Replaced all string-copying ones with ckmakmsg(). ckudia.c (18): Replaced all string-copying ones with ckmakmsg(). ckufio.c (26): There were quite a few holes in this one, OK now. ckupty.c (4): Replaced all string-copying ones with ckmakmsg(). ckuscr.c (1): Replaced all string-copying ones with ckmakmsg(). ckusig.c (0): OK ckutio.c (38): Lots of holes, OK now. Added ckctoa() routine to ckclib: character to string (substitute for "%c" printf format descriptor); it returns a pointer to a rotating buffer of 32 length-1 character strings. Thus using ckctoa(), ckitoa(), ckltoa(), ckitox(), etc, we can pretty much duplicate with ckmakmsg() any (s)printf() call that has four args or less and doesn't require padding. ckclib.[ch], 25 Jun 2000. Also added ckmakxmsg(), which is just like ckmakmsg() but takes 12 string args instead of 4. ckclib.[ch], 25 Jun 2000. Here's an example sprintf(): char tmp[64]; sprintf(tmp,"Initial value for \\&%c[%d]",x,v+1); /* Help string */ and its replacement, which requires two ckmakmsg() calls, or one ckmakxmsg(): char tmp[64]; int len; len = ckmakxmsg(tmp,64, "Initial value for \\&",ckctoa((char)x),"[",ckitoa(v+1),"]", NULL,NULL,NULL,NULL,NULL,NULL,NULL); Back to the modules... ckuusr.c (31): All sprintf's checked or replaced with ckmakmsg(), except the K95-specific one that constructs the "takepath" for the TAKE command, which needed to be replaced by a ckstrncpy() followed by 32 cknstrcat()'s, which not only makes it safe but also fixes a mismatch between the number of %s's (19) in the original sprintf() format string and the number (24) of string arguments. ckuus2.c (0): OK ckuus3.c (12): Replaced all unsafe sprintf's. ckuus4.c(247): Many \fblah() error message converted to ckmakmsg(). ckuus5.c (63): Mostly numeric; replaced all unsafe ones. ckuus6.c (44): Fixed many many holes in formation of dial strings. The original sprintf()s were left as comments for checking in case of dialing problems after "%s%s%s%s%s%s%s%s" transcription. Left domydir() directory-listing lines alone for the same reasoning given above for ckcfns.c. ckuus7.c (40): Tons of SAVE KEYMAP code. ckuusx.c (72): Most sprintf's were already checked. ckuusy.c (3): OK. Also in ckuus6.c: Fixed broken DELETE command in VMS (an #else was missing). The problem with "set host <1500 A's>" crashing C-Kermit was setlin() using strcpy() to write into a local array. I thought we had caught all those months ago... Fixed now. ckuus7.c, 25 Jun 2000. Cleaned up some of yesterday's changes. ckclib.c, ckuusr.c, 26 Jun 2000. Fixes from Jeff to typos in K95-specific portions of yesterday's changes. ckuus[r7].c, 26 Jun 2000. New functions for ckclib.c: ckuitoa(), ckultoa(), ckctox(). 26 Jun 2000. Combed thru ckuus7.c looking for strcat()'s and strcpy()'s -- there were dozens of strcpy()'s, only a few of them checked. Fixed them all. ckuus7.c, 26 Jun 2000. Similar treatment for ckuus[r3456xy].c, ckcfns.c, 26 Jun 2000. New ckclib.[ch] from Jeff, with const's added to ckmak[x]msg() input-string parameters to squelch complaints when used with Kerberos lib strings. 27 Jun 2000. strcpy/strcat treatment for ckcfn[23].c, ckcmai.c, ckucmd.c, ckudia.c, cku[tf]io.c, ckupty.c, ckuscr.c, 27 Jun 2000. Changed cvtdir(), the VMS function that converts BLAH.DIR;1 to [BLAH] (etc), to have a destination-buffer length argument, and to return the size of its result or -1 if the result doesn't fit the destination buffer or any other kind of error. ckvfio.c, 27 Jun 2000. Converted all cvtdir() references: ckuusr.h, ckuus[456].c, ckucmd.c, ckvfio.c, 27 Jun 2000. Discovered a problem in TRANSLATE for VMS: if a directory name was specified as the output file, but in BLAH.DIR;1 format, this wouldn't work. Fixed in doxlate() by using (new) length returned by cvtdir(), ckuus4.c, 27 Jun 2000. Changed APCBUFLEN definition not to refer to CMDBL, so APCBUFLEN can be used in modules (such as ckcfn*.c) that don't also #include ckucmd.h. ckcker.h, 27 Jun 2000. Discovered that cmcvtdate() would write into its argument if given a date in yyyymmdd format with no time. Fixed in ckucmd.c, 27 Jun 2000. From Jeff: strcat/strcpy/sprintf treatment for ckuath.c, ckcnet.c, ckctel.c, ck_crp.c, 28 Jun 2000. Cleaned up long lines & trailing blanks ckuath.c, ckcnet.c, ckctel.c, ck_crp.c, plus any other modules that had them, but not ckuath.c or ck_*.c. 28 Jun 2000. Tried building on SCO 5.0.5 and got some warnings on the new code, mainly char/CHAR. Added casts. ckuus[7x].c, ckcfns.c, ckctel.c, 28 Jun 2000. Compiling ckutio.c on SCO 5.0.5 fails because the timeval struct isn't defined any more, which is truly bizarre since absolutely nothing has changed in this area since 7.0 was released. Adding -DDCLTIMEVAL and -DNO_DNS_SRV fixed it but why didn't I have to do this before? And SOCKOPT_T has to be size_t. etc etc... Turns out to be a local problem; builds fine on a different 5.0.5 system elsewhere, false alarm, put the makefile entry back as it was. Some platforms (such as Windows) have a fixed program stack size (256K in Windows), therefore you can't use big automatic arrays in functions that might be invoked recursively, which we do in \fexecute(); this would make K95 crash after just a couple levels of recursion. The big buffers are vnambuf[] in zzstring() and line[] in doreturn(). Jeff changed the code to use malloc's for this. ckuus[46].c, 29 Jun 2000. We also had a potential problem with our homegrown printf replacement. From Jeff: I added some checks to the ckx[f]printf() routines and increased the size of the buffers we were using. The test is of the form: call [vs]printf() routine check strlen() of buffer and enforce a length smaller than the allocated buffer size. call doexit() if test fails. This will work in this case because the buffer being manipulated is not on the stack. It is important to note that if the buffer was on the stack it would be impossible to perform the test because immediately after the attack the instruction pointer would be set to point to the attacking code. The code as currently written should protect against the abuse of an attack as long as the placement of the buffers in memory does not allow the overwriting the memory pointed to by the instruction pointer stored in the call stack. ckutio.c, 29 Jun 2000. Also from Jeff: Additional code for handling filenames that contain spaces. ckclib.[ch], ckucmd.c, 29 Jun 2000. Corrections to above: keep previous brstrip() function since new one breaks the script programming language; add buffer-length argument to dquote() so we don't write past the end of a buffer (unless the caller is lying about its size). ckclib.c, ckucmd.c, 29 Jun 2000. Peter E made a convincing case that SET SERVER GET-PATH should not convert relative names to absolute. This surprises most people (e.g. if you include '.' in your Unix or DOS PATH, this means "my current directory at whatever time in the future I search the PATH", not "my current directory at the time the PATH was set"). NOTE: This is an INCOMPATIBLE change from 7.0 and earlier, but this should be acceptable since there was no way to do this before and you can still get the old effect by specifying absolute paths, or even by using \fpathname() on relative paths. parsdir(2): ckuus3.c, 29 Jun 2000. Peter E noticed that if you SET STREAMING OFF and then make a connection via a TCP/IP modem server, transfers stream anyway. This doesn't happen if you make an ordinary Telnet connection. I don't see how it can happen anyway. If you SET STREAMING OFF, this sets streamrq to 0 (SET_OFF); then when the protocol is started, streamok is set to 0, and then is set to 1 only if (a) it is negotiated, and (b) streamrq is SET_ON or SET_AUTO. Streaming is actually started by streamon(), which does nothing if streamok is not set. I do not see a hole here. But if I can reproduce the problem maybe I can find a hole. Since I don't have a TCP/IP modem server, the hard part is setting up a simulation using a Unix workstation in place of the modem server. SET HOST, SET MODEM TYPE, then CONNECT, start Kermit, CONNECT to the modem port with SET CARRIER-WATCH OFF, escape back to the original Kermit and DIAL, then CONNECT to the dialed-to system, log in, "kermit -r", escape back to the dialing system and send a file. Streaming is used, as expected. Then repeat the same procedure, but this time SET STREAMING OFF before sending the file. This works fine; the file is sent with windowing. Maybe it has to do with when the SET STREAMING OFF command is given. Repeat again, this time with SET STREAMING OFF done before the SET HOST. This works too. So I can't reproduce the problem. In any case, it occurs to me that we should not treat a dialed connection as reliable even though it was made over a TCP/IP connection. So in dodial(), I set reliable to OFF if DIAL or ANSWER succeeds. ckuus6.c, 29 Jun 2000. Somebody claimed that giving the -a command-line option before the -s option didn't work. It works fine. Fixed the program herald; it lost a space in the ckmakmsg() conversions. ckcmai.c, 29 Jun 2000. Peter E discovered that a very short binary file (7 bytes) could be spuriously identified as UCS2: xy. It should have been tagged binary because it contained a run of more than 2 NULs, but the only such run terminated at the end of the file, and the code missed that case. Fixed in scanfile(): ckuusx.c, 29 Jun 2000. Fixed ifdefs around Telnet-related keyword tables to be TNCODE rather than TCPSOCKET. ckuusr.c, 29 Jun 2000. Added \v(buildid), which is yyyymmdd of the current build, e.g. "20000629", useful mainly to developers and testers for whom the version number string and test ID strings are not fine-grained enough. ckcmai.c, ckuusr.h, ckuus4.c, 29 Jun 2000. Added code to the Unix CONNECT module to discard NUL after incoming CR on a Telnet NVT connection. This makes a difference for transparent printing. ckucns.c, ckucon.c, 29 Jun 2000. Jeff renamed the experimental version of brstrip() to fnstrip(). ckclib.[ch]. Also some changes to ckuath.c. 3 Jul 2000. Fixed a bug in ttlock() construction of the lockfile name introduced in the ckmakmsg() conversion; this one was in the HPUX-specific section and wasn't caught because I built & tested it with a non-ANSI compiler (the length arg was missing). ckutio.c, 3 Jul 2000. Fixed an external declaration of ttnproto for non-TCPSOCKET builds. ckuusr.c, 3 Jul 2000. Unix transparent printing, after the recent fixes, was working right, but the final "i" of the terminating escape sequence was being printed on the screen. Fixed in ckucns.c, 3 Jul 2000. Until now, CP1252 has been considered identical to ISO Latin-1, which is not the case since it puts graphics (e.g. "smart quotes", Euro symbol, etc) in the C1 area. Added a definition for CP1252 as a separate file character set in ckuxla.h. Added CP1252 to all the tables in ckuxla.c, but still using the Latin-1 translations. 3 Jul 2000. Added translations from L1, L2, LC, LH, LG, and Latin-9 to CP1252. ckuxla.c, 3 Jul 2000. Added table entries to the Unicode module. ckcuni.c, 3 Jul 2000. Added translations from CP1252 to L1, L2, LC, LH, LG, and Latin-9. ckuxla.c, 4 Jul 2000. "/NOBACKUP" is confusing -- it implies that something won't be backed up. Changed it to "/NOBACKUPFILES", meaning the action won't be performed on backup files (similar to "/NODOTFILES"). ckuus[r26].c, 4 Jul 2000. More squelching of warnings from ANSI compilers about whether a string arg is const or not. zinroot(), ckufio.c, 4 Jul 2000. More squelching of warnings from ANSI compilers about whether a string pointer is to chars that are signed or not. rlog_ini(), ckcnet.c, 4 Jul 2000. Peter E noticed that SET FILE SCAN OFF, DIR /X /TYPE:{TEXT,BINARY} didn't work. Fixed in fileselect(). ckuusx.c, 4 Jul 2000. See note from 29 June: "Peter E noticed that if you SET STREAMING OFF...". What he really meant to say was simply that if FILE DISPLAY was not FULL, the message said STREAMING when streaming actually was not being done. Fixed in ckuusx.c, 4 Jul 2000. Fixed problems that Jeff reported with ADD SEND-LIST: . GET while SEND-LIST defined didn't work. . CLEAR SEND-LIST, SEND caused a memory exception. ckuusr.c, 4 Jul 2000. Fixed matchname() (filename pattern-matcher) to strip pathname before performing the match. ckuusx.c, 4 Jul 2000. Peter E noticed that if you "set modem command init-string" (to nothing), Kermit still waits for a response (to nothing) and this stops dialing in its tracks. Fixed in ckudia.c, 5 Jul 2000. Peter E noticed that "kermit -Y -y blah" didn't execute "blah". Fixed in ckuusy.c and prescan(): ckuus4.c, 5 Jul 2000. Changed DELETE /ALL to DELETE /TREE (leaving /ALL available but invisible). ckuus6.c, 5 Jul 2000. There was some kind of confusion regarding the first arg (filename) to zstime() in HPUX ANSI C builds. For HPUX only, it was "const char *" rather than just "char *" for some reason, and now because recent changes (SET ROOT) we get warnings and failures galore with ANSI builds. I took the whole thing out and it builds fine. I wonder why I thought we needed the "const" business in the first place? ckufio.c, 5 Jul 2000. Several complaints appeared recently about Kermit hanging or going into infinite loops, but they are not necessarily related, since one involves IKSD (which has no CONNECT mode) and the others involve CONNECT mode. For CONNECT the culprit was this new bit from the transparent printing fixes: if ((c == NUL) && network && (ttnproto == NP_TELNET)) { if (prev == CR) /* Discard of if */ if (!TELOPT_U(TELOPT_BINARY)) /* peer not in binary mode */ continue; } Once this code is executed, conect() goes into an endless uninterruptible loop, eating all CPU time. The "continue" should have been a "break" -- it was continuing the wrong loop (remind me to get a 500-line screen). That should have fixed it, but it didn't. The problem was that obc (the screen output buffer counter) had already been incremented because of the NUL, but since we never output it, the next time through the loop we skipped the FD_SET() for the keyboard, and then found the same NUL and did it all over again, forever. Adding a call to ckcputf() to flush the screen output buffer before continuing the main loop fixed it. ckucns.c, 5 Jul 2000. So much for the CONNECT problem. For IKSD, we'll have to wait til next time we get a runaway... Back in April, I changed conbin(), concb(), etc, to remember the console state and not do anything if the state was already the desired one, thus avoiding expensive and possibly disruptive system calls when they were not necessary. But in shuffling the code, I neglected to include a return() statement at the end of conbin(), so it has been returning a random value all this time, resulting in "?Sorry, can't condition console terminal" failures at the beginning of CONNECT mode in some builds. Strange that all the picky compilers that complain so loudly about signed vs unsigned char didn't pick up on this one! Fixed in conbin(): ckutio.c, 6 Jul 2000. The SET DIAL TIMEOUT command was essentially ignored if it was less than 60 seconds, due to overconservatism. Better to do exactly what users tell us. Fixed and tested in ckudia.c, 6 Jul 2000. SHOW MODEM didn't remove the '*' from the AUTOANSWER ON string if the user changed it. Fixed in showmodem(): ckuus3.c, 6 Jul 2000. Peter E noticed that some modem commands were not reset if the user changed modem types and had previously given SET MODEM COMMAND commands to customize the previous modem type. These included Autoanswer On/Off, Ignore-Dialtone, Speaker On/Off, Volume, and Init2. Fixed in initmdm(): ckuus3.c, 6 Jul 2000. There has been a long-standing problem with canceling files during a streaming transfer. If you have a directory with lots of files, "send *" on a streaming connection, and type 'X' lots of times, eventually the transfer hangs. Packet logs reveal this happens if you hit 'X' just before the first Data packet of a file is sent. Logs taken of the same transfer on both ends both show that the last packet exchanged was the ACK to the A packet, i.e. receiver sent it and the sender got it, but then the first data packet is never sent and the sender never sends anything else either and since we're streaming, the receiver never times out. This suggests a problem in the protocol module. Capturing a case in a pair of debug logs, we see the 'X' is actually detected after the F packet is sent but before the ACK is received. At this point we go ahead and send the A packet and read its ACK, and after that we hang. Examination of the protocol state table shows no test for interruption in Y and Y states. Added them. This gets us to state, but we still hang. Now the receiver... Z state didn't account for interruption either; actually it was kind of a mess -- it didn't call reof() at all except by accident. It still doesn't but I added a sufficient workaround for this case. Finally, Y state didn't properly test the return code from sdata(). After fixing all of these, you can 'X' a streaming transfer as much as you like and it keeps chugging away until it gets to the end of the file list. However, 'Z' given at the sender still didn't work; this required a fix to Y state. Now all is well when interrupting the sender. ckcpro.w, 6 Jul 2000. Canceling individual files from the receiver is OK too, but canceling the batch does not stop the sender from continuing through to the end (even though the receiver refuses all incoming files after 'Z' is hit). The problem here is that the protocol for canceling a batch is for the receiver to put 'Z' in the data field of the ACK to a Data packet. But when streaming, Data packets are not ACK'd except when we need to send back an interruption notification, and in fact this works fine for per-file interruption but for batch interruption we didn't catch all the cases, in particular when the file can be fully transferred in one packet. Fixing this one required a sneaky extension to the protocol: allowing X and Z cancellation not only in ACKs to D packets but also in ACKs to Z packets. This won't hurt anything since prior Kermits ignore the data field of the Z packet. ckcpro.w, 6 Jul 2000. The hint for GET-Class Command Failed didn't mention the fact that for GET to work, the other Kermit has to be in server mode. Fixed in parser(): ckuus5.c, 6 Jul 2000. Yesterday's batch cancellation fix for GET had one minor flaw, namely that it would not work with preexisting servers. However, an E packet serves the same purpose, so now we try the graceful method once and if that fails, then the hammer. ckcpro.w, 7 Jul 2000. Now here's an awful bug we've had for the last 10 years or so... When IFDEBUG is defined then debug() is a macro that expands to "if (deblog) dodebug(...)". So what happens here: if (condition) debug(...); else return(-1); Answer: Not what you expect. This becomes: if (condition) if (deblog) dodebug() else return(-1); In other words, instead of returning (-1) when the condition is false, we return (-1) when the condition is true and not debugging. This could explain an awful lot of those bugs that go away when you "log debug". Now, I don't know how portable this is going to be, but I changed the IFDEBUG definitions for debug() and hexdump() from: #define debug(a,b,c,d) if (deblog) dodebug(a,b,(char *)(c),(long)d) #define hexdump(a,b,c) if (deblog) dohexdump((CHAR *)(a),(CHAR *)(b),c) to: #ifdef CK_ANSIC #define debug(a,b,c,d) ((void)(deblog?dodebug(a,b,(char *)(c),(long)d):0)) #define hexdump(a,b,c) ((void)(deblog?dohexdump((CHAR *)(a),(CHAR *)(b),c):0)) #else #define debug(a,b,c,d) (deblog?dodebug(a,b,(char *)(c),(long)d):0) #define hexdump(a,b,c) (deblog?dohexdump((CHAR *)(a),(CHAR *)(b),c):0) #endif /* CK_ANSIC */ (We can't use (VOID) because of the conflict with in ckuusx.c.) It never occurred to me before that you can put numeric constants in C programs as if they were statements, but I suppose it's no different from invoking a function that returns a numeric value, without assigning the return value to anything. Builds OK with both gcc and non-ANSI cc on Sun. We'll see how it goes in the next build-all. If there is trouble we can always go back to just calling dodebug() all the time. ckcdeb.h, 7 Jul 2000. Anyway, as awful as this one seems, constructions like the sample above are extremely rare. One of them, however, was in dogta() (the _GETARGS/_PUTARGS routine), which could affect FOR and WHILE loops, SWITCH statements, and IF {} ELSE {} constructions. Which is where I found myself while trying to figure out the following problem: The SHIFT command does not decrement \v(argc) if it is used within an IF { } or ELSE { } block, or in a SWITCH case. But fixing the debug() definition made no difference (it did indeed fix one problem with _PUTARGS, but that only uncovered a second one). It turns out there was also logic error in dogta() itself, in the section where _PUTARGS propagates the changed argument vector array, RETURN value, and \v(argc) back up the stack (it was doing this in the wrong direction -- another potential memory leak, but not such a horrible one, since it was writing into the correct array, but the wrong place in it -- the only "exception" would be if a FOR, WHILE, SWITCH, or IF was on the top of the stack AND and stack was full. dogta(): ckuus6.c, 7 Jul 2000. \fjoin() needed another option: to put quotes around any elements that contained spaces or that were empty (null). A third arg was added for this: 0 or missing means don't add quotes; nonzero means quote (enclose in doublequotes) any elements that contain spaces or are empty. ckuus4.c, 8 Jul 2000. In working on the previous item, I discovered that a loop like: for \%i 1 \fdim(&_) 1 { echo \%i. [\&_[\%i]] } adds 1 to the dimension of the argument vector array. Culprit: popclvl() forgot that the array dimension is one less than argc since array dimensions don't count element 0. Fixed in ckuus5.c, 8 Jul 2000. Back in Feb 1998 when I did all the work extending the macro arg list, creating the \&_[] array and the \%* variable, and then later added the SHIFT command, I chickened out of having SHIFT update \%* since by that time all the quoting and grouping info had already been lost. But now that we have the new \fjoin() function, we can use that to construct \%* on the fly from the current argument vector, no matter how much it's been shifted: \%* is: \fjoin(&_[],{ },1) The easiest way to handle this is in doshift(), which I did, works fine (a better, but riskier, approach would have been to eliminate the m_line[] array and replace all references to it by on-the-fly \fjoin() invocations). Anyway, now it's possible to have a recursive "argument eating" function like this: def xx { show args shift if ( > \v(argc) 1 ) xx \%* } Another shortcoming of \%* is that it wasn't available at top level; now it is, and SHIFT works on it too, even with quoted/grouped args. ckuus5,c, 8 Jul 2000. Discovered that although redefining \%2 also changes \&_[2], the opposite was not true. Fixed in addmac(), ckuus5.c, 8 Jul 2000. Similarly undefining \&_[1] did not also undefine \%1 (etc for 0..9), and vice versa. Fixed in delmac(): ckuus5.c, 8 Jul 2000. (I really should merge these into a single array some day...) Another possible shortcoming of \%* is that it doesn't reflect redefinitions of argument variables. If it's affected by SHIFT (as it is now) then it should also be affected by redefinitions. If anybody wants to refer to the original value, that's easy enough: just put "assign \%x \fcontents(\%*)" at the beginning of the macro before changing any of the parameters. zzstring() and fneval(): ckuus4.c, 8 Jul 2000. After all this, I went back and removed the m_line[] array, since now we always compute \%* on the fly when it is referenced. Good, one less thing to keep track of and one less potentially large array. ckuus[456].c, 8 Jul 2000. From Jeff: a fix to debug(F011,...) in which malloc() could be called with a negative argument, plus an authorization fix, ckuath.c, ckuusx.c, 9 Jul 2000. Also from Jeff: an experimental version of ttruncmd() that, instead of setting stdin/stdout of the new sub-process to the socket and then starting the subprocess and getting out of the way, starts the subprocess and enters a select() loop reading data from the socket and writing to the subprocess and vice-versa, applying Telnet protocol translations as needed. This is mainly to allow external protocols like Zmodem to work over Telnet connections. I'll hold off on taking this one until it's more finished. (See ttruncmd-select.c.) Jeff reported that the recent changes for file interruption broke attribute refusal. As noted, the Z state was a mess. Upon looking at it more closely it turned out to be not just a mess but totally nuts, so I rewrote it, hopefully with brain engaged this time, and also cleaned up reof() itself. First of all, I enclosed all the material relating to actually closing and handling the disposition of the output file in "if (o_isopen) { ... }" so we don't have to worry about calling reof() when a file wasn't actually opened. Second, I noticed that /RENAME-TO: and /MOVE-TO: failures had been ignored, but they should be fatal, so I fixed that (and added messages and transaction log entries for failures). ckcpro.w, ckcfns.c, 9 Jul 2000. From Jeff, a correction to SHOW MODEM for TCP/IP modem servers, some decryption corrections to Unix CONNECT mode, other minor things. ckuus3.c, ckucns.c, ckucon.c, ckuath.c, 10 Jul 2000. If the value of \%a is a string that begins and ends with doublequotes, then: echo \%a unexpectedly strips them. This is a wrinkle of the ECHO command. I changed ECHO to call brstrip() before evaluation, rather than after. Thus: echo "one two three" prints: one two three but: define \%a {"one two three"} (outer braces are stripped in the definition, as always) and: echo \%a prints: "one two three" ckuusr.c, 10 Jul 2000. The operation of \fsplit() was very confusing in its treatment of arrays that were already declared. At first I thought it might be nice to give the user control of the size of the array, but this leads to all kinds of craziness when calling \fsplit() repeatedly on different strings but using the same array -- old pieces left over from previous calls, new pieces having nowhere to go, etc -- all of which is documented, but not especially useful. So I changed \fsplit() to always create a new array (destroying any existing one first). ckuus4.c, 10 Jul 2000. If \fjoin() can turn an array into a string and preserve grouping, then \fsplit() should be able to do the opposite. But presently \fsplit() (and \fword()) ignore grouping, either with braces or doublequotes. So now let's make \fjoin() and \fsplit() into symmetrical functions such that after: \fsplit(\fjoin(&a[],{ }),&b[]) \&a[] and \&b[] are identical. To add grouping capability to \fsplit() and \fword() without breaking current behavior, I added an optional 5th argument, a grouping mask: 1 = doublequotes, 2 = braces, 4 = singlequotes, 8 = parens, 16 = square brackets, 32 = angle brackets; these can be OR'd together to make any number 0-63 (-1 is treated the same as 63). If a bit is on, the corresponding kind of grouping is detected. Thus if \%a is: a "b c" {d e} (f g) h i 'j k' [l m] p then: \fsplit(\%a,&a[]) = 16 \fsplit(\%a,&a[],,,0) = 16 \fsplit(\%a,&a[],,,1) = 15 ("") \fsplit(\%a,&a[],,,2) = 15 ({}) \fsplit(\%a,&a[],,,3) = 14 ("" and {}) \fsplit(\%a,&a[],,,7) = 13 ("", {}, and '') \fsplit(\%a,&a[],,,15) = 12 (etc...) \fsplit(\%a,&a[],,,31) = 11 \fsplit(\%a,&a[],,,63) = 10 \fsplit(\%a,&a[],,,-1) = 10 Doublequotes and braces are the only forms of grouping significant to Kermit; I put in the others because it was easy and somebody might find them useful. As yet there is no concept of quoting or nesting and I'm not sure there needs to be. Quoting would be devilishly difficult to use and explain. The main idea is to construct, deconstruct, and reconstruct arrays, which generally contain plain old up-front data, in which quoting rarely plays a role. ckuus4.c, 10 Jul 2000. Documentation note: If you include the same character in the grouping mask and the include list, the include list takes precedence. Example: def \%a a "b c d" e \fsplit(\%a,&a[],,,-1) = 3 <-- doublequote used for grouping \fsplit(\%a,&a[],,",-1) = 5 <-- doublequote not used for grouping Still do do: . Test . Check \fword(). . Add help text. . Give \fjoin() the same options for grouping (except not OR'd) Made dohexdump() int instead of VOID so new hexdump() macro definition would not result in "incompatible types in second and third operands of conditional expression". ckcdeb.h, ckuusx.c, 10 Jul 2000. In FreeBSD 4.1 CVSUP 7/10/2000 20:00 GMT-0500, endwin() now restores buffering of stdin (but not stdout), so now we can simply add -DCK_NEWTERM to the freebsd41 makefile entry, and everything works again. makefile, 11 Jul 2000. Added straightforward stack-based nesting support to \fsplit()/fword(), but it was too straightforward, needs more work, don't use it yet. ckuus4.c, 12 Jul 2000. Redid fsplit()/fword() from scratch as a simple FSA to support nesting. Added a new general purpose routine, cksplit(), to ckclib.c that does the work, thus removing a fair amount of code from fneval(). The new routine handles grouping if asked to, otherwise fnword()/fnsplit() behave as before. Grouping implies nesting. Any of the following pairs can be used for grouping in any combination: "" {} [] '' () <> with the restrictions that (a) a closer must match its corresponding opener, and (b) you can't expect to be able to use quotes for nesting. Up to 64 levels of nesting are allowed, and up to 4096 words in the result string. Example: a (b c n o) p is split up in any desired way according to the grouping mask. Malformed strings are handled appropriately, e.g. everything after an unmatched opener becomes the last (or only) word. ckcdeb.h, ckclib.[ch], ckuus4.c, 15 Jul 2000. Changed \fjoin() to accept the same kind of grouping mask as \fsplit() and \fword(), but in this case only the lowest bit value is used. ckuus4.c, 15 Jul 2000. Updated help text for \fword(), \fsplit(), and \fjoin(). And now \fsplit() and \fjoin() are "reciprocal" functions. You can split a string up into an array and join it back into a new string that is equivalent, except that the type of braces might change. Example: def \%a a {b c [d e] f g} "h i" j m echo STRING=[\%a] echo WORDS=\fsplit(\%a,&a,,,-1) show array a asg \%b \fjoin(&a,{ },2) echo JOIN =[\%b] echo WORDS=\fsplit(\%b,&b,,,-1) show array b The arrays a and b are identical. The strings a and b are as follows: \%a: a {b c [d e] f g} "h i" j m \%b: a {b c [d e] f g} {h i} j {k l} m Added a new module: ckcftp.c. Just an 825-line skeleton for now, containing command parsing and tables, definitions, etc, but no actions. Added built-in FTP and {SET, SHOW, HELP} FTP commands to main tables. Aside from this, all code specific to built-in FTP should be confined to ckcftp.c. Read comments at the top of the new module. ckuusr.[ch], ckuus[235].c, makefile, 15 Jul 2000. ideas... . Maybe point R-commands at FTP module if an FTP connection is active. . Give error in FTP action commands if FTP connection is not open. . Give error in non-FTP protocol-related commands if FTP connection open. . What about "set protocol ftp"? . Maybe add another CM_flag to disable non-FTP related commands. . Add switches. A couple minor corrections to ftp stuff from Jeff. ckcdeb.h, ckuus2.d, 16 Jul 2000. Discovered that ckitoa() and friends, even though they were designed to be called repeatedly, and use a circular buffer for returning results, could sometimes fail to insert the terminating NUL between result values, so constructions like printf("%s %s %s....",ckitoa(a),ckitoa(b),ckitoa(c)) could give junk. Fixed in ckclib.c, 16 Jul 2000. Fixed debug(F001,...) not to print odd-number-digit hex numbers (by calling ckitox). dodebug(): ckuusx.c, 16 Jul 2000. Back to character sets. We had a couple problems in the TRANSLATE command. First, it's only supposed to translate character sets, but it was also doing record-format conversion. Fixed in x[gp]nbyte(): ckcfns.c, 16 Jul 2000. Second, when translating from any character set that has C1 controls to Unicode, it failed to convert the C1 controls. This was a bug in all the blah_u() routines in the Unicode module, where "blah" is any ISO-2022 conforming 8-bit character set (the Latin alphabets, DEC MCS, etc). This one was actually true not just for TRANSLATE but also file-transfer. Converting in the other direction was OK. Anyway, now if you create a file with: main() { int i; for (i = 0; i < 256; i++) putchar((char)i); } containing all 256 possible bytes, you can convert it from Latin-1 to UTF-8 (or UCS-2) and back correctly. ckcuni.c, 16 Jul 2000. Added APL-ISO and APL-AIX terminal character sets to the Unicode module for use by any Unicode-based terminal emulator, but didn't fill in the tables yet. ckcuni.[ch], 16 Jul 2000. Added an optional second argument to \fcvtdate(), a format option number. Normally the result is yyyymmdd hh:mm:ss. Option 1 changes the date format to yyyy-mmm-dd, and option 2 to dd-mmm-yyyy (mmm = English 3-letter month abbreviation). ckuus[24].c, ckucmd.[ch], 16 Jul 2000. Filled in the APL tables: u_apl1[], u_apl2[], tx_apl1(), tx_apl2(). ckcuni.c, 17 Jul 2000. For docs: Unicode makes no distinction between ASCII ABCs and APL ABCs. A special set of "math" ABCs will be added to Plane 1 in Unicode 3.1 or 4.0, but they aren't there yet, and we're not supporting nonzero planes yet either. The recent changes to the protocol module broke non-streaming transfers. After the Z packet, we either hang (long file) or quit early without waiting for the B packet (short file). To reproduce: set streaming off set win 31 <-- or 20, etc. on both ends and transfer a file. The first problem is that the sender's log shows a missing block of ACKs. The culprit? Our famous call to ttflui() in input(): we flush the communications input buffers when the window size is 1 and we have just read the packet we want. But the window size isn't really 1; we only set it to 1 temporarily around the Z packet. The test in input() for window size should be against the negotiated window size (wslotn), not the current window size (wslots). This 1-letter change cures the ttflui() problem, which apparently has always been there, but the B packet still isn't sent. ckcfn2.c, 18 Jul 2000. As for the rest, to make a long story short, in state, I had changed the test of sdata()'s return code in early July from "== -1" to "< 1" while trying to fix a problem we were having with cancellation, which eventually was fixed a better way. But -1 means eof and 0 means "draining" -- two completely different things. Putting the test back as it was fixes the current problem and does not seem to reinstate the previous one. ckcpro.w, 18 Jul 2000. From Jeff: a fix to askmore() to prevent it from looping if Kermit is running as a TCP server and the connection drops. ckuusx.c, 18 Jul 2000. Minor fixes to typos & ifdefs from Jeff: ckuus[r5].c, 19 Jul 2000. \fword(1 2 3,4) returned an error instead an empty string. Fixed in cksplit(): ckclib.c, 19 Jul 2000. PeterE reported some lingering problems with client/server ops when switching between streaming and nonstreaming after yesterday's fixes. I couldn't reproduce them, but I was able consistently get the protocol to hang when giving an RDIR command over a nonstreaming connection. Same problem as yesterday... The sender is not sending the B packet. Same fix as yesterday, except in a different place. ckcpro.w, 19 Jul 2000. APL tables revised in light of newly discovered ISO-IEC/JTC1/SC22N3067, 1999-12-28. ckcuni.c, 20 Jul 2000. Noticed that SHOW CONNECTION was getting some of its fields confused. Reason: uidbuf[] was empty. I don't see why but in any case, I added some safety clauses to the connection logging code to fill it in if it's blank. ckuus3.c, ckudia.c, 20 Jul 2000. Added a safety clause to Unix sysinit() to make sure that uidbuf[] was set to something, in the worst case "UNKNOWN" if all methods of obtaining the username fail. ckutio.c, 20 Jul 2000. PeterE noticed that if you transfer a file using C-Kermit 7.1 locally and a recent version of C-Kermit on the far end, and then transfer another file using an old version of C-Kermit on the far end, the local Kermit does not clear the whoareu[] (and sysindex) values from the previous negotiation, and therefore might do some inappropriate automatic peer-recognition stuff. Fixed in ckcfns.c, 21 Jul 2000. PeterE suggested that "send *blah", when "*blah" succeeds in matching a filename, should repaint the field with the expansion. For example, if the directory contains one file containing the string "blah", e.g. "theblahfile" then "send *blah" should be repainted as "send theblahfile". This is important especially for command history/recall. It was easy enough to do since we were already doing it in VMS and K95; however, it didn't work for Unix filespecs starting with tilde. But the calls to tilde_expand() in cmifi() were redundant anyway, so I just removed them and all's well. So now "send ~/tmp/x" is repainted as "send /usr/olga/tmp/x". Ditto for all other commands that parse input-file names. (We still need the tilde_expand() call in cmofi(), however, since the tilde doesn't get resolved otherwise.) ckucmd.c, 21 Jul 2000. Fixed the runzero declaration in ckuusx.c to be outside #ifdef UNICODE. ckuusx.c, 21 Jul 2000. Disabled the built-in FTP code by defining SYSFTP in ckcdeb.h, 22 Jul 2000. Adjusted prototypes, ifdefs, etc, for VMS. ckcdeb.h, ckuusr.h, 22 Jul 2000. Built on Linux RH6.1/Intel with various configuration options: Full 1730K NOPUSH 1689K NOFTP 1730K NOCMDL 1695K NOUNICODE 1444K NORECALL 1726K NOCSETS 1362K NONET 1618K NOXFER 1352K NODEBUG 1543K NOLOCAL 1373K NOSHOW 1667K NOHELP 1484K NOCKXYZ 1725K NODIAL 1599K NOSERVER 1705K MINIDIAL 1704K NOIKSD 1683K NOSPL 1466K IKSDONLY 1408K NOICP 446K The gets() function can't be used; modern linkers refuse to link programs that use it. I replaced the call to gets() in readpass() with a getchar() loop: ckuus7.c. The NOCMDL version wouldn't link until I #ifdef'd out the call to setiks() in ckuus3.c; major surgery would be required to decouple SET IKS from the command-line module. Minor touchups required in: ckcdeb.h ckcfn3.c ckcfns.c ckcker.h ckcmai.c ckucmd.c ckudia.c ckuus2.c ckuus3.c ckuus4.c ckuus5.c ckuus6.c ckuus7.c ckuusr.c ckuusr.h ckuusx.c ckuusy.c. 22 Jul 2000. Added apl2, apl-plus, and apl-2741 character sets to the Unicode module. ckcuni.c, 23 Jul 2000. With the new file-scanning feature, there was no way to inhibit character-set translation for text-mode transfers, so I added SET TRANSFER TRANSLATION { ON, OFF }. ckuusr.h, ckcmai.c, ckuus[234].c, ckcfns.c, 23 Jul 2000. Added a /TRANSPARENT switch to SEND, MSEND, RECEIVE, and GET, disabling translation for this transfer only. ckuusr.[ch], ckuusx.c, ckcfn3.c, 23 Jul 2000. Looked into adding /TIMEOUT:n to SET HOST and friends. Conceptually it's simple, but in practice it's scary. First look in ckudia.c at what had to be done for dial timeouts (in VOS, OS-9, Amiga, etc). Then we might also need to clean up the TCP/IP operations, depending on how far they'd progressed -- e.g. the connection is open but Telnet negotiations are stuck, etc. Well, we already handle this (or not?) with Ctrl-C. Built full regular versions on assorted platforms: BSDI/OS 4.0: 1643K (Intel) FreeBSD 4.1: 1699K (Intel) SunOS 4.3.1: 1982K (Sparc, gcc) AIX 4.3.2.0: 2037K (RISC) Solaris 2.5.1: 2100K (Sparc, cc) HP-UX 10.20: 2311K (non-ANSI, non-optimized) SINIX 5.52: 2390K (MIPS) debug(F011,...) could write past the end of its buffer. Fixed in dodebug(): ckuusx.c, 24 Jul 2000. Fixed #ifdefs from Jeff for NOXFER vs SHOW CHAR, plus re-enable Kerberos ticket forwarding on one-way authenticated connections. ckuus4.c, ckuath.c, 24 Jul 2000 Added whoami() to OS-9 and Amiga versions. ck[i9]tio.c, 24 Jul 2000. Added code to VOS sysinit() to fill in uidbuf[]. ckltio.c, 24 Jul 2000. Looked at enabling the extra Unicode-only charsets in C-Kermit. Forget it; it would be a major rewrite. Worth doing, but later. From Jeff: Improved error handling for K95 disk-space checking. ckuus[r4].c, ckcfns.c, 25 Jul 2000. Bug of the day: Unix C-Kermit would dump core if you changed the system time while it was transferring a file (because of the time-left and CPS calculations for the f-t display). Fixed by telling it to ignore floating-point exceptions. sysinit(): ckutio.c, 25 Jul 2000. Fixed an unguarded reference to TELOPT_blah in ckucmd.c, 25 Jul 2000. In VMS, responding to the askmore() prompt with Ctrl-Z would throw Kermit into an infinite loop. Fixed by special-casing VMS to use raw binary i/o rather than "half-cooked" (otherwise EOF is returned on all subsequent input requests and Kermit exits). ckuusx.c, 25 Jul 2000. Added scanfile() call to TRANSMIT if no explicit file-type switches were included in the command. Also made sure that TRANSMIT skipped charset translation if TRANSFER TRANSLATION is OFF. ckuusr.c, 25 Jul 2000. Fixed brstrip() to heed COMMAND DOUBLEQUOTING setting. ckclib.c, 25 Jul 2000. Continuation of the filename-completion repainting work from a few days ago... Changed cmifi2() to supply doublequotes (or braces if COMMAND DOUBLEQUO OFF) around a filename obtained by completion if the name contains spaces. This works fine as long as completion is full, but partial completion doesn't work yet. Also we to handle the case where the user actually types an opening quote. And we need to handle braces as well as quotes. ckucmd.c, 25 Jul 2000. Previous item, cont'd... By simply advancing the pointer past the opening quote (or brace) -- in other words, totally ignoring it -- full completion works if the user begins the field with a quote or brace (that's where we left off yesterday). The trick is to remember that we did this, and then when repainting the field, we go back one extra space to write over the quote or brace that they typed with the one that was subsequently supplied anyway. Also, the dquote() routine had to be changed to work for both quotes and braces. Having done this, partial completion now works too, adding quotes or braces around if none were provided, and adding the matching closing quote or brace if an opening one was given. I thought about leaving off the closing one, but then if the partial completion stops after one or more trailing spaces, the user won't see it/them. This way they see trailing spaces, but they have to backspace over the closing quote or brace in order to continue. Anyway, now all forms of completion work with filenames that contain spaces. Directory names too. ckucmd.c, 26 Jul 2000. Similar treatment for ?-help. ckucmd.c, 26 Jul 2000. Similar treatment for output files (replaced do-it-yourself brace-stripping code by a call to brstrip()). cmofi(): ckucmd.c, 26 Jul 2000. Added SET TERMINAL TRANSPARENT-PRINT as in invisible synonym for SET TERM PRINT for agreement with some of the docs. ckuus7.c, 26 Jul 2000. Adding optional arguments to the TAKE command was a good idea but had a bad side-effect, namely that any arguments given replaced the current argument vector at the level where the TAKE command was given, and worse, the new values stayed in effect after the command file was complete, which could give nasty surprises. I finally hit on a clean way of fixing this: if the TAKE command has arguments, then instead of replacing the current argument vector and calling dotake(), we define a temporary macro containing "take ", and then execute the macro with the given arguments. ckuus[r2].c, 26 Jul 2000. After adding function diagnostics for C-Kermit 7.0, "if defined \fblah()" gave obnoxious messages for nonexistent functions. Changed IF DEFINED to have a special case for functions. Also now you no longer have to include any arguments: "if defined \fupper()" works ok; so does "if defined \fupper". doif(): ckuus6.c, 27 Jul 2000. SHOW CHARACTER-SETS could show garbage (or worse) after a SET TRANSFER CHARACTER-SET TRANSPARENT command. ckuus3.c, 27 Jul 2000. Implemented \v(exedir) for Unix. This might come in handy in case we want to allow for putting the system-wide init file together with the executable, which would certainly simplify finding it. ckuus[4y].c, 27 Jul 2000. HTTP changes from Jeff, mostly stripping leading spaces from messages. Plus more filling in of FTP module. ckcnet.c, ckcftp.c, 30 Jul 2000. CRECEIVE was broken, wouldn't parse. Fixed in ckuusr.c, 30 Jul 2000. Added { RECEIVE, GET } /PIPES:{ON,OFF} to override global TRANSFER PIPES setting for this command only. ckuusr.h, ckcmai.c, ckuus[26x].c, 30 Jul 2000. PeterE discovered that the autoupload string was broken. This happened during the great conversion sprintf->ckmakmsg conversion. Fixed in ckcpro.w, 30 Jul 2000. Allow the LOG command to accept a directory name, in which case the appropriate filename is supplied. ckuus4.c, 30 Jul 2000. Just for fun, added a rudimentary LISP S-Expression (SEXP) parser (cksplit() makes this easy). If a command starts with "(" it is interpreted as an S-Expression. Operators are: = (assignment), +, -, *, /, and ^. Numbers and variables are allowed, and as a special bonus, "long variable" (macro) names do not need to be enclosed in \m(...), so you can do: (= a 2) (= b 3) (= c (+ a b)) (* (+ a b) (- c a)) Of course you can also include regular variables and function invocations in the normal manner. S expressions print their result automatically if entered at top level. ckuusr.[ch], ckuus[235].c, 31 Jul 2000. Some NOXFER and Forward-X corrections from Jeff: ckcmai.c, ckuusr.c, ckctel.c, ckcnet.c, 1 Aug 2000. Minor improvements to S-Expression parser: . Added \v(sexpression) and \v(svalue). . Added HELP SEXPRESSION (HELP-only keyword). . Added SET SEXPRESSION ECHO-RESULT { ON, OFF, AUTO } . Fixed handling of negative exponents. . Better detection of invalid S-Expressions. ckuusr.[ch], ckuus[234].c, 1 Aug 2000. Dat Nguyen noticed a strange bug: asg str xxxxx echo \flen xxxxx zzstring() was checking for this (functions with no parens), but not reporting an error, and evidently the most recent string definition was sitting in just the right place. Simplified the logic by letting fneval() handle it. Also improved fneval() error messages to show the full proper name of the function rather than what the user typed. ckuus4.c, 2 Aug 2000. Just in case anybody wants to extend the S-Expression code to work with other data types (float, string), I changed dosexp() to return a string pointer to a string result, rather than an int. Also put the built-in ops into a keyword table so they don't have to be single characters; added SETQ, EVAL, and MOD as proofs of concept. ckuus3.c, 2 Aug 2000. From Jeff, add FORWARD option to TELOPT command, plus various auth changes. ckuusr.c, ckctel.c, ckuat*.[ch], 3 Aug 2000. Split out the floating-point formatting code from fneval() to a separate routine, fpformat(), and improved it by (somewhat crudely) changing results that ended in lots of 9's after the decimal point to increment the rightmost non-9 digit and lop the rest (but leave a trailing '0' to indicate this was done). So now, for example, \ffpadd(1.5,2.3) is 3.80 rather than 3.799999999999999 (as it is in, e.g. EMACS Lisp on the Sun). ckuus4.c, 3 Aug 2000. Added floating-point support to dosexp(). The result is floating-point if any of the operands are floating-point, and is formatted with the new fpformat() function (which also trims useless trailing 0's, as always), so the results are nicer than with most Lisps. As proof of concept, added built-in EXP, SQRT, LOG, and LOG10 functions. ckuus3.c, 3 Aug 2000. Discovered that "if " didn't work. Typo in boolval(): #ifndef FNFLOAT should have been #ifdef. ckuus6.c, 3 Aug 2000. Changed "=" to be a predicate rather than an assignment operator, for compatibility with Lisp; use SETQ for assignment. Added "<", "<=", ">", and ">=" as additional predicates. Predicates are operators that that return "0" if false and "1" if true (in Lisp they return Nil and T). ckuus3.c, 3 Aug 2000. Added ABS, MIN, and MAX, plus FP-to-Int conversion functions FLOOR, CEILING, TRUNCATE, ROUND. ckuus3.c, 3 Aug 2000. Broke creation of local variables out of dolocal() loop into a new routine, addlocal(). Had dolocal() call addlocal() in its loop. ckuus6.c, 3 Aug 2000. Added LET, which is like SETQ but defines a local, rather than global, variable. This is not exactly like Lisp LET, which creates the variable in the scope of the current S-Expression, but we can revisit later if anybody cares. ckuus3.c, 3 Aug 2000. Tried adding ?-help for S-Expression operators but didn't get it working. Fix it later. ckuusr.c, 3 Aug 2000. Problem: Typing a token (such as "!") at the prompt, immediately followed by a question mark, just says "one of the following: '!'", not very helpful. Ideally, we would get the same help here as we would if we typed a space before the "?". Did this in cmkey() "?" section by setting cmflgs == 3 ("?") before returning. Works fine if subsequent field is cmtxt() or cmfld(), but until now we never had a token followed by a keyword. Added code in cmkey2() to handle this, and -- if you call cmkey2() right -- it works, in that that it (a) lets you type "?" after the "(" and gives you a keyword list and then lets you continue from there; (b) editing, etc, work fine; and (c) the command still executes correctly whether you typed "?" or not. But it breaks command recall, at least in this case, plus it doesn't allow the CAR to be anything that's not in the operator table, so (e.g.) "(a)" is not accepted. So I put the parsing back to unhelpful cmtxt(). ckuusr.c, ckucmd.c, 4 Aug 2000. Changed S-Expression handler, if given a predicate as its outermost operator, to set SUCCESS or FAILURE according to its result. ckuus[r3].c, 4 Aug 2000. Added \fsexpression(), which takes an S-Expression as an argument (outer parens are not required) and returns its value. Normal quoting rules apply. ckuusr.h, ckuus4.c, 4 Aug 2000. Checked number of operands for each operator as in Lisp; table-driven via special keyword flags. ckuus3.c, 4 Aug 2000. Fixed SETQ to allow more than one pair of args, and to undef the variable if a definition is missing, and to return NULL if it has no operands (like real Lisp). ckuus3.c, 5 Aug 2000. Fixed "(setq a xxx)" not to assign 0 to a if xxx is not defined. ckuus3.c, 5 Aug 2000. The following: define baz { assign a XXX, return B } echo A\fexec(baz)C prints XBC instead of ABC. It turns out this problem, unlike the one similar one on May 30th, is entirely superficial and was fallout from the change to ECHO "foo" to keep the quotes. I put the code back as it was; the user has to type double doublequotes if she wants the quotes echo'd: echo ""foo"" "foo" ckuusr.c, 5 Aug 2000. The LOCAL directive had a flaw; when used for macro names, it only made the name local if a macro of the same name was already defined. Otherwise, the LOCAL name became global. Fixed in addlocal(): ckuus5.c, 5 Aug 2000. Fixed chknum() to return 0 if arg NULL or empty. ckclib.c, 6 Aug 2000. Changed \fjoin() to separate array elements with space by default. ckuus4.c, 6 Aug 2000. Fixed top-level IF command for numeric comparisons to do brace-stripping so "if = {\fsexpression(+ 1 1)} 2 " could be parsed. ckuus6.c, 6 Aug 2000. Fixed (+) (-) and (*) to return the same values as in Lisp (0, 0, and 1, respectively). ckuus3.c, 6 Aug 2000. Added S-Expression operators: FLOAT, IF, NOT, !=, AND, OR, ... ckuus3.c, 6 Aug 2000. Fixed CEILING and FLOOR to do the right thing if given integer args. ckuus3.c, 6 Aug 2000. Changed the size of floating-point formatting buffer from 64 to 1024, since nothing stops somebody from doing (^ 99999999999999999 99999999999999999999), and printf("%f") will merrily expand to any length needed). IEEE floating point on the Sun seems to stop at about 300 decimal places and then prints "Infinity", but elsewhere who knows). ckuus4.c, 6 Aug 2000. My clever strategy of doing everything in integer arithmetic until I saw a floating-point number was all wrong because it gives wrong answers if (a) the magnitude of any operand is greater than MAXINT (and there is no reliable way to test for that), or (b) a result would overflow MAXINT. Therefore, all computations have to be in floating point anyway, and therefore there's no point in having all the #ifdef CKFLOAT's inside the S-Expression code, so I removed them all. ckuusr.h, ckuus3.c, 6 Aug 2000. Updated help text. ckuu2.c, 6 Aug 2000. Updated 7.1 release notes. ckc71.txt, 6 Aug 2000. Built on Sun with -DNOSEXP. It's only 8K smaller than with. TRUNCATE, FLOOR, and CEILING didn't truncate. Fixed in ckuus3.c, 7 Aug 2000. Changed fpformat() to be in #ifdef CKFLOAT rather than #ifdef FNFLOAT. ckuus4.c, 7 Aug 2000. Initialized malloc'd INPUT match buffer to the empty string. Previously it was not initialized; new strict malloc implementations are deliberately writing junk over malloc'd memory, rather than 0's, to force the application to initialize. ckuus4.c, 7 Aug 2000. Changed SHOW MACRO to accept a series of macro names rather than just one, and to only do an exact match on each one unless it contains metacharacters, in which case it does a pattern match, and also to omit blank lines. So now "show mac a b c" is a convenient way to check the results of operations like (setq a 1 b 2 c 3). ckuus5.c, 7 Aug 2000. ARRAY LINK . Makes a symbolic link from one array to another. Any changes to the link affect the real array. Deleting the link affects only the link. Deleting the array deletes all links to it. Resizing a link resizes the real array. Resizing a real array resizes all links to it. This mechanism allows arrays to be passed as parameters to macros. ckuus[r25].c, 8 Aug 2000. Added UNDECLARE as a top-level command, since we never had a straightforward way of undeclaring an array. ckuusr.[ch], ckuus2.c, 8 Aug 2000. Fixed \fdefinition() to require an exact match. ckuus4.c, 8 Aug 2000. Moved setup of PI and E constants to initfloat(), ckuus4,c, 8 Aug 2000. Added PI, SIN, COS, and TAN to S-Expression parser. ckuus3.c, 8 Aug 2000. Fixed a typo that broke -DNOFLOAT builds. ckuusr.h, 8 Aug 2000. More conditionalization to allow successful compilation and linking on platforms where we have floating-point arithmetic but can't (or don't) link with the math library. ckuus[34].c, 9 Aug 2000. Added extensibility to S-Expressions by allowing Kermit macros as operations. ckuus3.c, 9 Aug 2000. Made "*", when used as an operand, behave as in Franz Lisp: it's the value of the last S-Expression (1 initially). So, for example, (+ * *) executed repeatedly gives 2, 4, 8, 16, ... ckuus3.c, 9 Aug 2000. Added IF INTEGER as synonym for IF NUMERIC. ckuus6.c, 9 Aug 2000. From Jeff: Minor fixes to RLOGIN and TELNET command error handling and RLOGIN /K5 processing. ckuusr.c ckuus7.c, 10 Aug 2000. Fixed Motorola SV88, FreeBSD, NetBSD, and OpenBSD entries to include -DFNFLOAT and -lm, and built OK on those platforms. Reminder for next time we have trouble with this stuff: to get the f.p. math functions, both -DFNFLOAT and -lm must be included in the makefile entry. FNFLOAT is not defined automatically. makefile, 10 Aug 2000. Fixed Motorola SV68 makefile entries to include -DFNFLOAT and -lm (verified by Gerry B). makefile, 11 Aug 2000. Looked into making the algebraic expression evaluator, evala(), automatically lookup macro names like the S-Expression evaluator does. No dice. It can't work because evala() doesn't require spaces. Macro names like "a+b" and "foo&bar" are perfectly legal, but they would be ambiguous in algebraic expressions if we allowed bare macro names. ... But maybe if enclosed in parens? ... But then it might as well be an S-Expression or \m(xxx). ... But it can't be an S-Expression. Anyway, this doesn't work either because the algebra parser is strictly left-to-right. Case closed. 11 Aug 2000. Changed \v(exedir) setup in Unix to convert to fully qualified pathname in case argv[0] is relative. Also for platforms where argv[0] is not a full pathname, I added code to search the PATH for the executable. In all cases, it verifies that the result exists. In case all of this fails, it sets \v(exedir) to "/". 11 Aug 2000. S-Expressions: Added &&, ||, ** as synonyms for AND, OR, and ^. Added &, |, and # for bitwise AND, OR, and XOR. ckuus[23],c, 11 Aug 2000. addmac() did not return consistent return codes for array errors -- not declared, subscript out of range, etc. Fixed in ckuus5.c, 11 Aug 2000. S-Expression parser appeared to allow assignment to undeclared or out of range array elements. Fixed in ckuus3.c, 11 Aug 2000. If a command started with "{" but wasn't properly terminated, Kermit would recurse until stack overflow. Fixed in docmd(), ckuusr.c, 12 Aug 2000. Unix zfnqfp() was supposed to include a "/" at the end of a directory name, and it used to, but this was lost during the conversion to realpath() in April 1999. Fixed in ckufio.c, 12 Aug 2000. Noticed that ?-help inside of quoted filename didn't work any more. Typo (trashed comment terminator), fixed in cmifi2(). Also removed all the #ifdef OLDHELP code -- we haven't used it in years. ckucmd.c, 12 Aug 2000. cksplit() did not guard against too-deeply-nested strings. Fixed in ckclib.c, 12 Aug 2000. Added some small optimizations to mlook() and mxlook() (macro table lookups) to speed up scripts and especially S-Expressions. ckuus5.c, 12 Aug 2000. Same optimizations added to xlookup() (which is used for SEXP operators). ckucmd.c, 12 Aug 2000. Since the SEXP operator table is an xlookup() table, which need not be in alphabetic order, I rearranged it to have the most common ops at the top: SETQ, +, -, *, /, etc. ckuus3.c, 12 Aug 2000. After this a 5000-iteration test loop full of S-Expressions went about 15% faster, not as good as I hoped. gprof shows we're spending only 2% time in xlookup() and mxlook() (good) but still > 37% in regular lookup(), plus nearly 10% in cksplit() (not surprising). Lookup reports 33% cache hits. Close inspection of lookup() does not reveal any further possible optimizations, except maybe using a binary search if the table is large. So the real question is: can we reduce the number of calls to lookup() in the first place? Yes: In cmkey(), we can check FIRST whether we have a token, then call lookup() only if we don't. Trying this reduces our lookup() calls dramatically for a loop full of S-Expressions, raises the lookup() cache hit rate to 80%, moves lookup() down in the profile from 37% to < 4%, and increases the speed improvement of our test case from 15% to 36%. ckucmd.c, 12 Aug 2000. While I was at it... After the above, docmd() was pretty much the top hog. Rearranged it a bit to put the most common commands at the top, to reduce the number of comparisons after top-level command lookup. This didn't make any perceptible difference in execution time, but in the profile it reduced docmd() from 11.5% to 9.6% of wall time. The top hogs now are cksplit() and gtword(), but there's not much we can do about them. ckuusr.c, 12 Aug 2000. More script optimization: It's very common in a script to change the value of a macro -- especially when using SEXPs. What happens when we do this: define foo 1 define foo 2 the first DEFINE looks it up and, assuming it wasn't already defined, inserts it into the macro table. The second DEFINE looks it up, finds it, deletes it (freeing the name the value and moving everything past it up one slot), then moves everything down one slot, and allocates the name again and inserts the new value in the same place it was before. Obviously when *replacing* a macro definition, we can skip most of that. Changed addmac() to do so. Now the test script runs 39% faster. ckuus5.c, 12 Aug 2000. Squeezed another few percent out by doing a pre-xlookup() check in dosexp(), so now it's 42.5% faster. ckuus3.c, 12 Aug 2000. Added code to actually record a histogram of top-level commands. To view it, "log debug" just before exiting, then "grep CMSTATS debug.log", and look up the command indices in ckuusr.h. Found a couple more frequently used commands that needed to be moved up in docmd(), moved them, and now the test script is 47% faster. Hmmm, almost 50... So now cksplit() dominates the profile... A bit more micromanagement (inlining ckstrchr(), fixing isfloat()'s return code to indicate whether the argument was an integer or floating-point, thus obviating additional calls to chknum() or ckstrchr(), ...) brings us up to a 51% improvement. ckclib.c, ckuus3.c, 12 Aug 2000. Jeff noticed some fallout from all the optimizations, mainly that labels are sometimes parsed incorrectly; the error is "Not confirmed". This occurs when the label name is left behind after the label parse; when cmcfm() is called, gtword() returns the label name again. This problem goes away when I undo the "don't call lookup() if it's a token" optimization. It turns out that the token-handling code in cmkey() and in ckuus5.c had never been used before now; the lookup always succeeded because the tokens are also in the main lookup table, so cmkey() never actually returned -5 ("it's a token") before. But unfortunately, this was by far the biggest optimization, so undoing it is not exactly satisfying. The new problem has something to do with ungword() (cmcfm() still sees the label name in the buffer), but moving or removing the ungword() call (or adding an unungw() call at the obvious place) only makes matters worse. Backing off on the offending optimization fixes the problem but brings us back down to a 23% improvement. But I thought of another way to optimize lookup() internally, based on the fact that it requires its command lists to be in order, allowing an early loop exit, and this brings us back up to 49%. But now keyword lookups fail if the table is out of order, whereas previously they might have succeeded. The new rule is: all letters are counted as lowercase. Therefore characters like ^ and _ come *before* them, not after them. I had to rearrange the top-level keyword table plus a couple of others before the demo script would run again. ckucmd.c, 14 Aug 2000. Added code to lookup() to print a big message if the debug log is active and a table is out of order, so in case we notice any other funny behavior, we can just "log debug" and see if a TABLE OUT OF ORDER message comes out. Corrected a bug in the dosexp() prelookup optimization; it was testing the index for flags, rather than the flag word pointed to by the index. ckuus3.c, 14 Aug 2000. Corrected a bug in macro-name lookup optimization, in which suddenly case mattered. ckuus5.c, 14 Aug 2000. OK, back to the real problem... When I first added tokens (mainly to allow shell escapes without a space between "!" and the shell command), it required a kludge in which a space was inserted between the token and the next field in the internal buffers (#ifdef KLUDGE in ckuus5.c). Well, now that I fixed it so the token-handling code is actually executed instead of just sitting there being skipped over, we don't need the kludge any more; commenting out the #define KLUDGE statement seems to fix everything, and now the improvement jumps to 54%. And today's speedups apply to all compute-bound scripts, not just those that are dominated by token-initiated commands, so it wasn't a wasted day. ckuus5.c, 14 Aug 2000. Before leaving lookup(), added one more optimization: if the search object was not in the cache and the keyword table size is greater than 4, we do a quick binary search to find the first table element whose name starts with the desired letter (or higher). This, plus the other optimizations, boosts performance of non-S-Expression compute-bound scripts by up to 100%, reducing the number of lookup() loop iterations (and therefore string comparisons) to between 0.5 and 4 per lookup() call, depending on the application (that's down from an average of around 50). Added the same code to mlook() and mxlook(); it doesn't hurt, but it doesn't help much either, at least for small test cases. But it should make a difference for large programs containing many macro names. ckucmd.c, ckuus5.c, 15 Aug 2000. OK, back to S-Expressions. First, SETQ and LET didn't work right when given more than one assignment pair and the object of the second or subsequent pair was a backslash thing (\%a, \&a[1], etc). There was a hideous hack in dosexp()'s caller that peeked at the op to see if it was LET or SETQ, and then doubled the backslash (if any) of the second "word". So: (setq \%a \%x) would become: (setq \\%a \%x) But this didn't work for: (setq \%a \%x \%b \%y) However, just looping through the string doubling backslashes of every second operand doesn't help, because SETQs can be nested: (setq \%a (+ 1 (setq \%b 0))) But looping through the string and looking for items following "setq" or "let" doesn't work either because that would miss the final two terms in a construct like this: (setq \%a (+ 1 (setq \%b 0)) \%c \%x \%d \%y) In general: (+ 1 (setq \%r (+ 2 (setq a (setq \%x \%b))) x 4 \%y 5 z (+ \%a 1))) the only way we can tell the target of an assignment from the material to be assigned is by executing the S-Expression. And that means we can't pass ANY S-Expression through zzstring() at top level, since we don't know yet whether it contains any SETQs or LETs. Therefore, dosexp() has to call zzstring() on each and every piece of every S-Expression individually, and poof, there goes our good performance. The only way to salvage what we have so far is to insist that if the target of an assignment is a backslash variable, the backslash must be doubled by the user. So: (setq \%a (+ 1 (setq \%b 0)) \%c \%x \%d \%y) must be entered as: (setq \\%a (+ 1 (setq \\%b 0)) \\%c \%x \\%d \%y) ckuusr.c, 15 Aug 2000. Except for SETQ and LET, S-Expression operators were not being recognized if given in uppercase. Fixed in xlookup(): ckucmd.c, 15 Aug 2000. Fixed broken IF command (a couple keywords were misplaced). ckuus6.c, 16 Aug 2000. Problem: (setq \%i (setq \\%i 1)) failed to fail. Reason: \%i is not defined, so "setq \\%i 1" became the name of the variable to be SETQ'd, but since no value was given it was undefined. But since it wasn't defined in the first place, nothing happened. To catch this I had to add another check on the SETQ/LET target: fail if it has multiple words. Luckily this doesn't drag performance down too much. ckuus3.c, 16 Aug 2000. Feeling adventurous, I tried changing the algebraic expression evaluator (evala()) to treat non-numbers as macro names, just as S-Expressions do now. 10 minutes later... Hmmm, that was too easy. The demo script still executes OK, and there's no performance penalty. The change is in gettok(): #ifdef COMMENT..(new code)..#else..(old code)..#endif. ckuus5.c, 16 Aug 2000. Now you can use macro names ALMOST anywhere you can use a number: in any cmnum() field, as function arguments, as array subscripts, etc. How about as FOR-loop variables? All I had to do was comment out a check against it in dofor(): ckuus6.c, 16 Aug 2000. Fixed the algebraic expression evaluator not to spit out multiple error messages for one bad expression. ckuus5.c, 16 Aug 2000. Added ++ (increment) and -- (decrement) operators for S-Expressions. So now you can do (++ a) instead of (setq a (+ a 1)). You can also include a value, e.g. (++ a 2) or (-- a 3.14). You can also increment/decrement multiple variables in a single expression. ckuus[23].c, 16 Aug 2000. Added CHMOD /TYPE:{TEXT,BINARY} One-upsmanship on Unix; this lets you do things like "chmod /files /type:binary 775 *", i.e. only give execute permission to binary files. Also CHMOD /SIMULATE, which lets you see what would happen without actually doing it. ckuus[23].c, 17 Aug 2000. More-prompting didn't work in the DELETE command; del_pag initialization was wrong. Fixed in ckuus6.c, 17 Aug 2000. Added DELETE /TYPE:{TEXT,BINARY}. ckuusr.h, ckuus[26].c, 17 Aug 2000. Added GREP /TYPE:{TEXT,BINARY}. ckuus[26].c, 17 Aug 2000. The change on 21 Jul 2000 to repaint the field when completion was done on a filename that contained a metacharacter broke recognition of UNIX tilde-notation for directory names. Fixed in cmifi2(), ckucmd.c, 17 Aug 2000. Fixed another buffer vulnerability. "kermit -x xxxxxxxxxxx..." (more than 128 x's). ckuusy.c, 17 Aug 2000. Made SUCCEED and FAIL invisible to make top-level ?-menu fit on 24x80 screen again. ckuusr.c, 17 Aug 2000. From Jeff: Allow compilation when Kerberos is defined without rlogin: ckuus4.c Work on Heimdal support. NetBSD and SuSe have decided to ship Heimdal Kerberos as part of the base package instead of MIT Kerberos. IBM announced yesterday that all Netfinity servers will come with SuSe Linux pre-installed side by side with any other OS. ckuath.c, 18 Aug 2000. Changed a "(int) = (float)" assignment to avoid a warning. ckuus3.c, 18 Aug 2000. Fixed S-Expressions to handle the situation where only one word is contained in the parentheses, and it is the name of a macro whose definition is an S-Expression. ckuus3.c, 18 Aug 2000. Fixed S-Expression parser to catch unbalanced expressions up front rather than by recursing, and to disallow S-Expressions as operators, and to handle "trivial nesting" like ((((((((+ a b)))))))) better, and to ignore spaces between parentheses. ckuus[35].c, 18 Aug 2000. Yesterday's directory-name parsing fix wasn't quite right: "cd ~user" worked, but would back up too far and wipe out part of the prompt. Fixed in cmifi2(), 18 Aug 2000. Changed (EVAL) to allow any number of operands (0 or more) rather than requiring exactly one, and made "." a synonym for it. This provides a kind of grouping (useful in IF expressions). ckuus3.c, 18 aug 2000. Fixed evaluation of empty S-Expressions to not dump core, and added the built-in symbolic constant NIL. ckuus3.c, 18 aug 2000. Added built-in constant T. Now that we have three of them, put constants into a table so we can (a) look them up and (b) disallow assignments to them. ckuus3.c, 18 aug 2000. Increased maximum number of items in an S-Expression from 32 to 1024 if BIGBUFOK defined (as it is on modern platforms). ckuusr.h, 18 aug 2000. Changed evala() to accept floating-point numbers, truncating them to integers. ckuus5.c, 18 Aug 2000. Fixed a bug I introduced a few days ago when making algebra gettok() accept a macro name in place of a number, in which we could dump core if a word was given that was not found in the macro table. Also I discovered there was still a way it could give multiple error messages for a bad number, and fixed that too. ckuus5.c, 19 Aug 2000. After the last couple days' additions, the test suite runs about 2.9% slower, not so bad. gtword() was now #2; gave it a brief going-over. The main change was to put the entire editing section (character/word/line deletion, redisplay, etc) within "if (at-top-level)". This gains us another few percent in the heart of the command parser. ckucmd.c, 19 Aug 2000. The cmdsrc() function (which tells whether the command parser is at top level or parsing from a file or a macro) was being called more than almost any function, so I reworked the whole business to use a global variable, xcmdsrc, which is set whenever the command level or source changes, and replaced all calls to cmdsrc() by a simple reference to xcmdsrc. The function is still there, however, in case it is called from any other modules. This change made the test script run several percent faster. ckuus*.c, ckucmd.c, ckcfns.c, ckcpro.w, 19 Aug 2000. The scary thing now is makestr(); in our S-Expression benchmark it is called 14 million times: 7 times more than gtword() and 4 times more than setatm(). That seems really excessive. It's because of cksplit() and dosexp() -- they both call it on every word, and then once more to free each word. First I tuned makestr() a tiny bit more, replacing strlen() and strcpy() by tight inline loops, which helps a wee bit. Then in cksplit(), we need to not call makestr() when we don't have to. So the word pointer array gets a parallel length array. Whenever we want to add a word to the array, instead of calling makestr(), we check if the new length is less than the current one, in which case we just copy over the previous one. Otherwise we call makestr() and set the length word to what makestr() created (makestr() was altered to set a secret global variable to the length of the string it created so we don't have to do a redundant length counting). Anyway, in our benchmark the number of calls to makestr() from cksplit() was cut about in half, and the improvement goes up to 57% (from 54% a few days ago, and about 51% yesterday). Take out the profiling and it's 74%. ckclib.c, 19 Aug 2000. Prior to this change our benchmark was calling makestr() 14 million times. Now it's still calling it 13 million times. That's because dosexp() itself calls makestr() everywhere it needs to set its return value. Changed dosexp() to copy the result into a static internal buffer (on the call stack) using a tight loop (rather than [ck]str[n]cpy()). Oddly enough, this cuts out only another 1.5 million makestr calls. But the non-profiling improvement is up to 76%. All this will need a lot of testing to make sure I didn't break anything. ckuus[3rx].c, 19 Aug 2000. Yesterday's changes broke evaluation of () again. Fixed in cksplit(): ckclib.c, 20 Aug 2000. Discovered that \ffpraise() had its test for domain errors backwards: a negative number can be raised to a integral power, but not to fractional power. Fixed in fneval(): ckuus4.c. Same fix made in dosexp(): ckuus3.c, 20 Aug 2000. Fix typo in b_to_u() (Byte-to-UTF8) when returning UTF8 version of U+FFFD, from Bruno Haible. ckcuni.c, 21 Aug 2000. More SEXP profiling... In our test case, makestr() is now called 11.8M times: 0.00 0.00 121/11852872 xwords [73] 0.00 0.00 364/11852872 addmac [23] 0.08 0.00 500060/11852872 docmd [5] 0.23 0.00 1450159/11852872 setword [30] 0.35 0.00 2201168/11852872 parser [3] 0.46 0.00 2900299/11852872 cksplit [9] 0.75 0.00 4800520/11852872 dosexp [7] [18] 4.8 1.86 0.00 11852872 makestr [18] setword() calls makestr() 1.4M times but setword() itself is called 2.6M times: [30] 1.5 0.37 0.23 2650290 setword [30] so the strategy of recycling malloc'd buffers is paying off as expected, cutting the malloc()/free() burden for splitting by about 50%. Now cksplit() and dosexp() need some help. The remaining calls in dosexp() are unavoidable since we have to make (and free) a local copy of the word list for reentrancy. But calling makestr() just to free a temporary local variable is overkill, so the makestr(&blah,NULL) calls at the end can be replaced by simple free()'s. This brings the total makestr() calls down to 9.4M. And then where we make a pokeable copy of the argument string, we do the same trick of creating the buffer only if it doesn't exist yet, or if the argument string is longer than the current buffer, reducing makestr() calls to 6.5M. ckclib.c, 21 Aug 2000. But cksplit() still dominates the profile. There is always room for more optimization... The next trick is to get the length and copy into the working buffer in the same operation. Only rarely will the buffer be too small (in which case we quit the loop early, get a new buffer, and start over), but in most cases we accomplish the copy AND get the length in one very tight loop. This brings our performance improvement to 80% (up from 76% on Saturday). ckclib.c, 21 Aug 2000. Discovered that I broke macros-as-opcodes when (a) the macro is given operands and (b) does not include a RETURN statement. Fixed in ckuus3.c, 21 Aug 2000. Defining a macro as an S-Expression which is a predicate didn't work if the result was False. Fixed in ckuus3.c, 21 Aug 2000. Part of the optimization involved having dosexp() return "" rather than NULL sometimes, but in a couple places, I was still checking its return code only for NULL. This prevented certain error checks from working. Fixed in ckuus3.c, 21 Aug 2000. A couple more tiny optimizations to dosexp(): 82%. ckuus3.c, 21 Aug 2000. Fixed a problem with "sho mac partialname partialname...". ckuus5.c, 22 Aug 2000. Fixed another problem with NULL return values from dosexp() and added some more small performance tweaks. ckuus3.c, 22 Aug 2000. While looking at addmac(), noticed a lot of unnecessary work searching thru the macro table, so added the well-known speedups. ckuus5.c, 22 Aug 2000. Squeezed a few more cycles out of lookup() by eliminating strlen() calls. ckucmd.c, 22 Aug 2000. Eliminated 15% of isfloat() calls by going thru a macro, xxfloat(). ckuus3.c, 22 Aug 2000. More tweaking of cmtxt(), gtword(), ungword(). With these and previous tweaks we suddenly shoot up to 162% improvement (the test loop which originally took 16.0 seconds to execute now takes 6.1 sec). ckucmd.c, 22 Aug 2000. Fixes for HTTP command (give error message when operands missing) from Jeff. ckuusr.c, 22 Aug 2000. Added \v(ftime) which returns the time of day in seconds since midnight, including a fractional part. ckuusr.h, ckuus4.c, 22 Aug 2000. And another optimization for dosexp(): if the argument, after stripping any outer parens, is less than three characters long, there's no need to call cksplit(), since it can't possibly be split. This reduces cksplit calls in our test from 1.40M to 0.85M. And in this case we don't have to call makestr() either, which gets rid of another 0.5M makestr() calls, and brings the improvement up to 171%. ckuus3.c, 22 Aug 2000. Updated HELP FUNC JOIN text to describe n2. ckuus2.c, 23 Aug 2000. Fix from Jeff for typo in cksplit() resulting in reference to uninit'd variable xx. ckclib.c, 23 Aug 2000. Security & ftp updates from Jeff: ckcmai.c, etc, 23 Aug 2000. Fixed one of my last-minute optimizations from last night that broke cmtxt(....,NULL) case and therefore clobbered macro definitions. 23 Aug 2000. Jeff noticed that ckitoa() didn't work right when called lots of times. Fixed in ckclib.c, 23 Aug 2000. Fix from Jeff for SET FLOW ? message. ckuus3.c, 23 Aug 2000. There was trouble with S-Expressions which called macros which were themselves defined in terms of S-Expressions, when those macros had args. In this case dosexp() has to call itself on each macro arg to get the arg evaluated, and then copy the evaluated args to a buffer that holds the macro's argument list, so we can pass it to dodo(). The problem was this buffer wasn't on the stack. But you can't just put it on the stack because Windows doesn't like automatic buffers in routines that might recurse deeply. So we have to malloc and free the buffer at each invocation, so S-Expressions that call macros are not going to be very efficient. ckuus3.c, 23 Aug 2000. Added single-word Lisp-like quoting for macro arguments only. If an arg is prepended by a single quote ('), it is not sent to dosexp(), but passed literally to the macro. This allows (e.g.) macro names, operators, etc, to passed as arguments to macros -- kind of like function pointers. ckuus3.c, 23 Aug 2000. From Jeff: move misplaced #endif in gtword() that broke autoupload at command prompt: ckucmd.c; ftp progress: ckcftp.c, ckuusr.h, ckuus[45x].c. Updated SSL modules. 24 Aug 2000. Added a QUOTE operator to the SEXP parser (for grouping multiple words / character strings as macro arguments). Programming: easy. Explaining: hard. ckuus3.c, ckc71.txt, 24 Aug 2000. Discovered that \fsplit(), if called repeatedly, would dump core. That's because some of this week's optimizations pulled the rug out from under it (it used to "move" pointers from the cksplit() result array to the created array; now it has to create copies of each string.) ckuus4.c, 25 Aug 2000. Implemented Lisp group-quote shorthand notation: (a '(b c d)), equivalent to (a (quote (b c d))). ckclib.c, ckuus3.c, 26 Aug 2000. Discovered that addlocal() was making use of global buffers for intermediate results, which is not good for recursion. Also, it was doing WAY too much work due to lazy programming. Added a new routine, vardef(), to return a pointer to the value of any user-defined variable, given its name (\%a, \%1, foo). No more zzstring(), no more needless copying, ... This should speed up any macros that have LOCAL statements. ckuus5.c, 26 Aug 2000. A while back in the gtword() optimizations, I had clobbered the user's ability to edit when typing at an ASK prompt. Fixed in ckucmd.c, 26 Aug 2000. In the "allow doublequotes for grouping" changes, I overlooked cmnum(). Fixed in ckucmd.c, 26 Aug 2000. \m(x) returned the definition of any macro whose name started with x, as long as it was the only one. Although it might be dangerous to change this, it's even more dangerous to leave it as is, since it can return value of a different variable than the one you think you're asking for. Fixed in zzstring() by calling mxlook() rather than mlook(). ckuus4.c, 26 Aug 2000. Another nice touch for SEXPs: If the last statement in a macro is an S-Expression, its value is returned automatically. ckuus5.c, 27 Aug 2000. Macro setup blindly called delmac() 10 times, once for each argument. I changed it to call delmac() only when it needs to. ckuus5.c, 27 Aug 2000. One final problem for SEXPs. Currently the dosexp() return value is in a static internal 32-byte buffer on the call stack. This should be big enough to hold any reasonable number, and still allow recursion to any reasonable depth (like 1000) without filling up memory & swap space, and without any special performance penalty. But then again, what's reasonable? Anyway, now that we have QUOTE, it seems unreasonable to limit its operand to 32 bytes. But making the buffer much bigger would be bad for recursion. The alternatives are: (a) make it a little bit bigger, like 80 bytes; or (b) keep a stack of return values, which we malloc as needed. But (b) would mean we have to place some limit on the recursion depth in order to know how big to make the stack, and that we have to malloc storage for each return value. So there goes performance and flexibility. So I chose (a) and raised the maximum result length to 79. Anybody who needs to work with numbers longer than that probably should be using some other package, and the QUOTE limitation can be circumvented by using ' instead of QUOTE. We can do (b) if we ever come back and add string and list processing. ckuus3.c, 27 Aug 2000. The performance improvements gained last week degraded a bit over the last few days: 171% down to 148%. I trimmed away a few excess debug() calls to little effect. Performance is still quite good so let's leave well enough alone. Anyway, new improvements that aren't measured in the test case. From Jeff: fix Telnet handling of duplex variable, which controls echoing: ckuus7.c, ckctel.c. Auth changes: ckcdeb.h, ckuath.c. More FTP: ckcftp.c. 28 Aug 2000. In the "accept macro names as numbers everywhere" change, I overlooked the floating-point arguments of the \ffpxxx() functions. Fixed in fneval(), ckuus4.c, 28 Aug 2000. Added CHECK SEXPRESSION. ckuus3.c, 28 Aug 2000. Updated HELP SEXP. ckuus2.c, 28 Aug 2000. Built full (but non-crypto, non-profiling) version on Linux/i386. Size: 1777167. Built with -DNOSEXP, had to add an #ifdef to doexit(): ckuusx.c. Size: 1758383. Total size of SEXP code: 18K. Also built a NOFLOAT version, which required an #ifdef in gettok(): ckuus5.c. 28 Aug 2000. Peter E discovered that C-Kermit, when receiving an empty file, no longer bothers to create it (ditto for K95). A chunk of code seems to have disappeared from the protocol module's Z state some time after C-Kermit 7.0 was released. I copied the exact same code back from 7.0 and all is well. ckcpro.w, 29 Aug 2000. Peter E pointed out that IF VERSION, though documented in the book, was never actually implemented. Added it. ckuus6.c, 29 Aug 2000. Fixed some signed/unsigned conflicts in ckvfio.c reported by Mark Berryman. 29 Aug 2000. Made a first pass over the new ftp module; reformatted, fixed long lines, removed trailing whitespace, sorted out and fixed prototypes, made some progress at removing ANSI C dependencies but didn't finish. Note definition of closesocket() at the top of the file. This function is called all over the place but not defined anywhere. Seems to build and run OK. ckcftp.c, 29 Aug 2000. FTP updates from Jeff. ckcftp.c, 30 Aug 2000. Updated HP-UX makefile entries from Peter E, 30 Aug 2000. Changed UNIX ztime() to accept a NULL argument, in which case it only sets the global ztmsec and ztusec variables. ckutio.c, 30 Aug 2000. Added a cheap version of gmstime() to ckcftp.c. 30 Aug 2000. Dat Nguyen discovered that "echo \fsexp(foo)", where foo is the name of a macro that returns a string, returns junk under certain (well, most) circumstances. The sequence is: ECHO "ABC \sexp(foo) XYZ" The ECHO command calls cmtxt() to parse the echo string. cmtxt() calls zzstring(). Now zzstring() is copying ABC to atxbuf; when it hits \fsexp(foo), it calls fneval(). fneval() sees that it's \fsexp(), and so calls dosexp(). dosexp() sees it's a macro and calls dodo() and then parser(). parser() calls cmtxt() to evaluate the RETURN value. But this trounces over the buffer zzstring() was using. So dosexp(), when it encounters a macro, has to do exactly what \fexecute() does. namely cmpush()/cmpop() around parser() execution. Of course this is HUGELY expensive, so I added an up/down counter in \fsexp(), so dosexp() only does cmpush/pop() if \fsexp() is active. ckuus3.c, 30 Aug 2000. While I was chasing this down I also changed dosexp()'s method for returning its result to the more general one of malloc'ing a buffer of the needed size, no matter how big -- no more artificial size limit on what can be returned; using the "only malloc if new value is bigger than previous one" method, the performance hit isn't too bad. ckuus3.c, 30 Aug 2000. But having done that, there was no more barrier to supporting string operands and results, so I took the few extra steps to enable them: . Store string results internally enclosed in '(). . Translate (QUOTE (a b c)) to '(a b c) so the two forms are consistent. . When fetching the value of a macro and it starts with ', don't run it. . Some other stuff like that. Now we can do just about anything with strings that we could previously do with numbers: use them as operands, store them in variables, pass variables having string values as arguments to functions, etc. ckuus3.c, 30 Aug 2000. Noticed that the trick I did for the ECHO command (allowing 'echo "foo"' to work as before) wasn't right, so fixed it. ckuusr.c, 30 Aug 2000. From Jeff: ckcftp.c corrections; ckuus7.c updates to show authentication; ckuath.c updates for show authentication; optimizations to reduce the number of DNS queries that are required for Kerberos 4/5 authentication; ckcnet.c - correction to SRV and TXT record queries. 31 Aug 2000. As expected, some minor repairs were needed after yesterday's SEXP changes. First problem: I broke (setq x) (i.e. assignment of nothing to a variable). Fixed in dosexp(), ckuus3.c, 31 Aug 2000. Fixed up SET SEXPRESSION DEPTH-LIMIT and added SHOW SEXPRESSION and revised update notes for recent S-Expression changes. ckc71.txt, ckuusr.h, ckuus[r35].c, 31 Aug 2000. Fixed an unguarded reference to oldplex variable in settrm(): ckuus7.c, 31 Aug 2000. From Jeff: KRB5_U2U #ifdefs added + other auth changes. ckcnet.c, ckuath,c, ckutio.c, ckcdeb.h, etc. 1 Sep 2000. The change I made to UNIX ztime() to allow construction of a gmstimer() routine for FTP (a) had a typo, and (b) didn't cover all the cases. Fixed in ckutio.c, 1 Sep 2000. It seems xlookup() calls cklower() on its argument, thus lowercasing it IN PLACE. This had the side effect of lowercasing SEXP '(Strings). Worked around in dosexp(): ckuus3.c, 1 Sep 2000. Fixed xlookup() not to lowercase its argument. This gives a slight performance hit, but it's the right thing to do. Btw, the only clients of xlookup() were S-Expressions and Kverb lookups. ckucmd.c, 1 Sep 2000. Discovered that my addmac() optimization from last week had a bug that could cause core dumps under certain infrequent conditions. Fixed in ckuus5.c, 1 Sep 2000. Added ckstrrchr(), ckstrpbrk(), and ckstrstr() to ckclib.[ch], and changed FTP code to use them instead of their nonportable non-ckstrxxx() counterparts. Also replaced references to other strxxx() routines with calls to ckstrxxx() for other functions that were already in ckclib.c, like ckstrchr. Also replaced sprintf()'s by ckmakmsg() calls or direct assignment. Made all routines static that didn't have to be referenced from outside. Made all function declarations and other code independent of ANSI C and finished the internal prototypes. ckcftp.c, 2 Sep 2000. Fixed up SHOW FTP command a bit (don't display SYSFTP stuff if NEWFTP defined, show current connection, if any). ckuus5.c, ckcftp.c, 2 Sep 2000. Made "ftp " a legal command (i.e. "open" can be omitted). ckcftp.c, 2 Sep 2000. Added "ftp bye" as invisible synonym for "ftp close". ckcftp.c, 2 Sep 2000. Changed the BYE command itself to close the FTP connection if (a) one was open, and (b) no other connection was open. ckuusr.c, ckcftp.c, 2 Sep 2000. Put in the FTP PUT/APPEND switch parsing, but most of them don't have any effect yet. ckcftp.c, 2 Sep 2000. Integrated MPUT with PUT and APPEND and made it work. If MPUT was the command, and/or if the filespec was wild, we call gnfile() in a loop just like in Kermit. This brings in the full range of file selection (size, date, etc), and sending from a file-list file works too. Got group interruption working. ckcftp.c, 2 Sep 2000. Implemented [M]PUT /DELETE and /MOVE-TO:. ckcftp.c, 2 Sep 2000. Implemented automatic text/binary mode switching during PUT. The hierarchy is: if a /TEXT or /BINARY switch was given, that's the mode we use; otherwise, if FILE SCAN is on, we use that to pick the mode; otherwise if a SET FTP TYPE command was given, we use that; otherwise we use the prevailing SET TYPE TYPE. ckcftp.c, 2 Sep 2000. Summary: the following (should) all work now: FTP PUT /AFTER: FTP PUT /BEFORE: FTP PUT /NOT-AFTER: FTP PUT /NOT-BEFORE: FTP PUT /BINARY FTP PUT /TEXT (= /ASCII) FTP PUT /DELETE FTP PUT /MOVE-TO: FTP PUT /DOTFILES FTP PUT /LARGER-THAN: FTP PUT /SMALLER-THAN: FTP PUT /LISTFILE: FTP PUT /NOBACKUPFILES FTP PUT /NODOTFILES FTP PUT /RECURSIVE (sort of) FTP PUT /TYPE For FTP PUT and APPEND, there is one source filespec (which may be wild); for FTP MPUT there can be any number of source filespecs (like Kermit MSEND). Notes at top of ckcftp.c are updated. Some corrections from Jeff to yesterday's work: ckcdeb.h, ckutio.c, ckcftp.c, 3 Sep 2000. Added username to SHOW FTP display. ckcftp.c, 3 Sep 2000. Added SET FTP FILENAMES { LITERAL, CONVERTED }. For now the default is LITERAL (but maybe it should automatically change to CONVERTED if peer is not UNIX?) Implemented filename conversion if FTP FILENAMES is CONVERTED or if PUT /FILENAMES:CONVERTED is used. ckcftp.c, 3 Sep 2000. Added SET FTP PATHNAMES { RELATIVE, ABSOLUTE, OFF }; default is OFF for now. Implemented pathname handling for this, with override by PUT /PATHNAMES if given. Previously the pathname was left on, but the server rejected incoming files whose names contained paths. ckcftp.c, 3 Sep 2000. Implemented PUT /RENAME-TO:xxx, like SEND /RENAME-TO. Added transaction log entries for PUT. ckcftp.c, 3 Sep 2000. New FTP code from Jeff in support of MGET. ckcftp.c, 4 Sep 2000. Moved oldplex declaration outside of #ifdef NETCONN since now we seem to be using it for other things. ckuus7.c, 4 Sep 2000. Fixed ptransfer() to do its CPS calculation in floating point. It was dropping fractions of seconds from the elapsed time. ckcftp.c, 4 Sep 2000. Discovered that the brief-format transaction log format was broken in the strncat-to-ckstrncat transition; the time field was left blank. This is because the ckstrncat() length argument is the *buffer* length, not the number of chars to copy. Did a census of ckstrncat() calls (there are about 360 of them), and found the following: ckcnet.c: ckstrncat(name,ckuitoa(ntohs(saddr.sin_port))); ckcnet.c: ckstrncat(name,ckuitoa(ntohs(saddr.sin_port))); How do these even compile??? (I didn't touch them.) All the other calls look OK. Fixed and modernized doxlog(), and adapted FTP to use it if SET TRANSACTION-LOG BRIEF. ckuusx.c, 4 Sep 2000. Gave fpformat() the circular-buffer treatment (like ckltoa() & friends) so it could be called repeatedly, e.g. in a function argument list (like printf, ckmakmsg, etc). ckuus4.c, 4 Sep 2000. Changed ptransfer() (even though we're not going to be using it later) to print numbers in regular notation (by calling fpformat()) rather than ugly scientific notation. ckcftp.c, 4 Sep 2000. SET TRANSACTION-LOG FTP, LOG TRANSACTIONS (which uses the IKSD logging code instead of dotlog()) didn't work at all. It didn't even work in original 7.0. Not only did it not work, it corrupted the source file when sending by appending the FTP log entry to the end of the source file! (Only if you gave those commands, not when done with --xferlog/--xferfile) It's a good thing C-Kermit has so many obscure features that nobody noticed this one. Fixed in traopn(): ckuus4.c, and in doiklog(): ckufio.c, 4 Sep 2000. Did not implement the FTP-format transaction log for FTP because it is all done in zclose(), so we'll get it automatically when/if we convert the present file i/o to use the Kermit APIs. Commented out the .netrc processing; I don't think we want it. ckcftp.c, 4 Sep 2000. Added connection switches for scripting, to avoid nonscriptable Name: and Password: prompts: FTP [ OPEN ] [ switches ] [ ] Switches are: /USER:username /PASSWORD:password /ANONYMOUS /ANONYMOUS supplies user "anonymous" and the appropriate @ as password. openftp(): ckcftp.c, 4 Sep 2000. From Jeff: New hexdump(): ckuusx.c; USE_RUSERPASS #ifdefs for ckcftp.c; corrections to host:port-building strncat's in ckcnet.c, 5 Sep 2000. CLOSE TRANSACTIONS was never coded to handle the FTP-format transaction log. Fixed in doclslog(): ckuus5.c, 5 Sep 2000. Discovered that the FTP-format log entries always said "o" (for output file) even if it was an input file, because I shadowed the zclose() "n" (file ID) argument when calculating the length of the record probably in the June 2000 sprintf() rampage. Fixed in zclose(): ckufio.c, 5 Sep 2000. dotlog() was clobbering tralog() if called when TRANSACTION-LOG format was set to FTP, so subsequent SHOW LOG commands would say the log was closed when it wasn't. Fixed in dotlog(): ckuusx.c, 5 Sep 2000. Dat Nguyen noticed that (eval a b c ...) stopped after the first operand. Fixed in dosexp(): ckuus3.c, 5 Sep 2000. While chasing Dat's reported bug, noticed that floating-point ops were spewing out "log10: DOMAIN error" messages for any floating-point operation that involved a negative number. That's from yesterday's change to fpformat(). Fixed in fpformat() by taking the log of the absolute value of the number. ckuus4.c, 5 Sep 2000. From Jeff, ckcdeb.h ckcnet.c ckcftp.c, 6 Sep 2000: Added missing HTTP Header to the HTTP commands to support virtual hosts. Finally found out why FTP AUTH KRB4 was not working in all builds: . On Windows, there was a bad bug in the KRBV4W32.DLL library which prevented all SAFE and PRIVATE communications from taking place after FTP AUTH KRB4 was negotiated. . On all platforms that use the old K4 distribution there is a conflict in the format of the DES Key Schedule which prevents the krb_mk_priv() / krb_rd_priv() functions from being used to generate PRIVATE messages. These routines perform their own DES manipulations using an externally provided Key Schedule. If the Key Schedule was generated from another library with a different format, the computations will be incorrect. Therefore, it is not possible to support FTP AUTH KRB4 if the old MIT K4 distribution is being used AND either OpenSSL or Eric Young's DES library are in use. This is now enforced by #defines at the top of ckcftp.c . Fixes for security feedback messages when making FTP connections: ckcftp.c. More FTP work, ckcftp.c, 6 Sep 2000: . Filled in FTP PWD command. . Implemented verbose mode and made it the default. . Fixed FTP to restore default username before opening a new connection instead of reusing the one from the previous connection. . Added FTP PUT /UNIQUE to override global FTP UNIQUE setting. . Fixed gmstimer() -- it was totally wrong, results were complete garbage. . Added authtype to SHOW FTP. . Fixed openftp() parsing code to handle editing correctly. Jeff noticed that "http /header:{xxx : ?" gave strange results. After a wild goose chase prompted by the colon (which is used as a break character when parsing a switch), I realized the problem was more subtle, and has always been with us. When parsing a field, if it is enclosed in braces, we do not break on space. But if ? is typed, we go back and reparse starting at the point where we left off. But then we lose the brace count, and therefore the next space entered (after the ?) causes the field to break prematurely. Fixed by adding code to recalculate the brace level upon entry to gtword() from whatever happens to be sitting in the reparse buffer. Ditto for doublequotes. ckucmd.c, 6 Sep 2000. Corrections from Jeff to yesterday's changes. ckucmd.c, ckcftp.c, 7 Sep 2000. Created a new symbol FTP_SECURITY, which is defined if any of the security methods is selected, and put all security-related keywords, keyword tables, and code within it. This is because the keyword and switch help menus and parsing could be rather confusing on non-secure builds; e.g. "set ftp authtype" had only one keyword, "automatic", which was invisible. Rearranged "show ftp" to list security methods, and not list security options that were not available. This might need some refinement. ckcftp.c, 7 Sep 2000. Made doexit() call ftpbye() if there is an open FTP connection to shut it down gracefully. ckuusx.c, 7 Sep 2000. Added FTP QUOTE (invisibly, just for us to use while developing). Added account field to FTP USER command, plus /ACCOUNT: switch for openftp(), not that it will ever be needed. No time to test any of this. ckcftp.c, 7 Sep 2000. From Jeff: SET AUTHENTICATION KERBEROS5 ADDRESSES: ckuusr.h, ckuus[237].c, ckcnet.c, ckuath.c; Corrections to yesterday's work: ckuusx.c, ckcftp.c, 8 Sep 2000. Peter E sent in a couple of Unix Compress (.Z) files that scanfile() misdiagnoses as UCS-2 BE. Since there is no BOM, the diagnosis is because the number of 0 bytes in even positions is about 10 times the number in odd positions. But: . The total number of 0 bytes is only about 2% of all bytes. . The numbers of 7-bit and 8-bit bytes are approximately equal. Lesson: In the absence of a BOM, we should be very careful about declaring a file to be UCS-2. At the place where we are about to declare it UCS-2: . If decision is based on ratio of 0's in even/odd positions, require a higher percentage of 0 bytes, like 20%? . If distribution of bytes is fairly uniform, it's probably compressed. In that case, don't call it UCS-2. . If file is short, there's not enough evidence, call it binary unless there is no doubt. Did all this in scanfile(), and also weighted the other factors in the non-BOM UCS-2 case more heavily against UCS-2 since it's always better to send a UCS-2 file in binary mode than to send a binary file in text mode. ckuusx.c, 8 Sep 2000. Dat wanted to be able to use S-Expressions in the operator position of an S-Expression; I changed the code to allow it. Also, dosexp() was returning a bogus value when evaluating empty parentheses. ckuus3.c, 8 Sep 2000. \fsplit() was broken for non-SEXP uses, another casualty of the optimization blitz of a couple weeks ago. Fixed in cksplit(): ckclib.c, 8 Sep 2000. Added top-level command VOID, which is just like ECHO except it doesn't echo. Can be used (e.g.) to invoke functions that return values without actually doing anything with the value: "void \fsplit(\%a,&a)". ckuusr.h, ckuus[r2].c, 8 Sep 2000. Added FTP TYPE (sends TYPE command to server, like other FTP clients, unlike FTP SET TYPE, which just sets the variable locally). Also added FEAT and OPTS commands from RFC 2389, though we don't actually do anything with the responses yet. ckcftp.c, 8 Sep 2000. Fixed FTP CHMOD, which wasn't sending a properly formed command to the server. ckcftp.c, 8 Sep 2000. Moved ftpbye() call in doclean() outside of the "if (local)" clause, since opening an FTP connection does not set the "local" variable (and doing so would be dangerous, since all code presently assumes that if local is nonzero, this indicates a device or connection was opened by ttopen() and must be closed by ttclos()). ckuusx.c, 8 Sep 2000. Tested most other commands except MGET, MDEL, and friends; all seem OK. Corrections from Jeff to yesterday's work, plus auth changes: ckuath.c ck_ssl.c ckcftp.c ckcnet.c ckcmai.c ck_ssl.h, 9 Sep 2000. Implemented dynamic as-names for FTP, e.g. "put * \v(filename).new". ckcftp.c, 9 Sep 2000. Changed \v(ftp_msg) to not include the numeric code, since \v(ftp_code) already has it (if they agree, there's no reason to reflect it in both variables; if they don't, why don't they?). ckuus4.c, 9 Sep 2000. Added SET FTP PROGRESS-MESSAGES { ON, OFF } to control the "if (testing)" messages. ckcftp.c, 9 Sep 2000. Fixed some unguarded references to Unicode-specific variables in scanfile(): ckuusx.c, 11 Sep 2000. A few characters (:, $, ?, etc) were still missing from the cksplit() nosep arg in dosexp(), preventing use of variables containing those characters in SEXPs. Fixed in ckuus3.c, 11 Sep 2000. Dat complained that you can't have an SEXP operand that is an SEXP whose operator is an SEXP. I started on this one by trying: def cat return \%1\%2\%3 (setq abc 23) ((cat '+ '+) abc) ; works (increments abc) (++ (cat 'a 'b 'c)) ; fails The "works" part was done on Friday. I fixed the failing case, and its generalizations, in the code section that applies only to SETQ, LET, ++ and --. But unexpectedly, this seems to have also fixed the case that Dat was reporting even though it did not involve those operators, no doubt through the magic of self-reference. ckuus3.c, 11 Sep 2000. Fixed FTP IDLE when given without a time. ckcftp.c, 11 Sep 2000. Made FTP MDIRECTORY invisible because I don't see any difference from DIRECTORY, nor any reference to it (MLIST?) in the RFCs. ckcftp.c, 11 Sep 2000. Added the missing pieces to FTP PUT for automatic text/binary mode switching. ckcftp.c, 11 Sep 2000. Started looking at recursive PUTs in FTP. Began by fixing SET FTP PATHNAMES and FTP PUT /PATHNAMES to use the right keyword table. Then added a very crude sort for the file list, using a fixed-size array, just as a prototype (in #ifdef SORTNAMES); we need the list sorted to manage directory-changing. Seems to work fine, adds no noticeable delay for reasonable numbers of files, but of course it soaks up storage, and will also introduce a delay for large numbers of files. However, we only need to do this when PATHNAMES are not OFF, which they normally are, in which case the presort is skipped. Second step: made the array dynamic, but still fixed size (the limit is arbitrarily set at 10,000 files). ckcftp.c, 11 Sep 2000. Dat reported another problem with a complex script involving S-Expressions that use macros as operators. I totally rewrote the SEXP debugging code so I could follow better what was going on in the log. The problem boils down to this: dosexp() gets an operator it doesn't recognize, so it looks it up in the macro table and gets its index. Then it loops through the operands, calling itself recursively to evaluate them. But should any of these operands happen to be SEXPs whose operators are macros, there is always the possibility that they might alter the macro table. So at the end of the argument-evaluation loop, the index of the macro to be executed might no longer be valid. So I added code at the point where we invoke the macro to first compare the given macro name with the name of the macro at the index we found when we looked it up; if they differ, look it up again. ckuus3.c, 12 Sep 2000. Back to FTP... Noticed that the transaction log says "status ok" even for files that failed (e.g. because they include a pathname). Reason: ftp_put("STOR","foo/bar.txt","foo/bar.txt") returns 1, even though the reply is "553 foo/bar.txt: No such file or directory". Turns out to be because ftp_put() never checked sendrequest()'s return code and always returned 1. ckcftp.c, 12 Sep 2000. Fixed FTP getreply() to trim trailing junk (usually just a ^M) from the end of ftp_reply_str[] so we can use it in messages, and changed FTP code to include failure reason (ftp_reply_str[]) in transaction log entries. ckcftp.c, 12 Sep 2000. Now back to recursive PUTs... Having the FTP code build a list before the fact and sort it is not practical, because MPUT can produce a list of lists, in which each element is a wildcard that can expand to up to MAXWLD names (typically 102400), and there is no practical limit on the number of elements. Therefore we might easily run out of memory when trying to combine them. So instead, let's try sorting each list. This is done quite simply, by inserting a call to sh_sort() in nzxpand(). ckufio.c, 12 Sep 2000. From Jeff: Remove #if[n]def NEWFTP from all files. It was only added to allow the compilation of the FTP code since we had temporarily defined SYSFTP by default in ckcdeb.h. NEWFTP is not supposed to be necessary to use built-in FTP support. Fixed IKS Timeout parameter check. Changed parameter for NTLM validity check. NTLM now prompts the user for the username, domain, and password. For use when logging into IKSD on NT or Microsoft Telnet Server. You can now login as someone other than the local account. The work of the past few days: . C-Kermit / Kermit 95 now compatible with the forthcoming OpenSSL 0.9.6 release. Compatibility is maintained with 0.9.5. . Implemented SSL/TLS session reuse: Telnet and HTTP connections Peter Runestig is working on code to allow session reuse between Kermit as client and multiple simultaneous telnetd/ftpd/iksd sessions to the same host. . More work on providing support for Heimdal Kerberos 5 (NetBSD, OpenBSD, FreeBSD, and SuSE Linux.) Code compiles but getting weird errors. ckcdeb.h ckuath.c ckuus7.c ckuusy.c ckuusx.c ckuusr.c ckuus5.c, 13 Sep 2000. doftpput(), which parses all the FTP PUT switches and then sends the files, was getting too big. In the file-sending loop, I moved the code that sends each file out of the switch case and into a separate routine, putfile(), since we have to add lots more code here for directory switching, and many C compilers choke on switch cases with lots of code. ckcftp.c, 13 Sep 2000. Gathered repetitive code into little routines: doftpcwd(), doftpmkd(). ckcftp.c, 13 Sep 2000. Got rid of SET FTP PATHNAMES, the PUT and GET /PATHNAME switches, and the corresponding variables (ftp_pth, x_pth). ckcftp.c, 13 Sep 2000. Got rid of SORTNAMES code. ckcftp.c, 13 Sep 2000. Added syncdir() routine for use by PUT. Outbound filenames are all passed through this routine, which does any needed CWDs, MKDs, and CDUPs, and then the filename is sent to the server without the path part. All directory changing and creating is done one segment at a time, using only the segment name and no separators or delimiters, so we can do recursive PUTs to servers on UNIX, VMS, VOS, Windows, etc. Directory changing and creation commands are kept to a minimum by the presort that was done on the file lists, but we don't actually depend on the presort -- it still works without it, but with more client/server CD/MKD overhead. The entire directory tree is sent and replicated, except that (a) directories that contain no files are not replicated, and (b) symlinks are not re-created (just like with recursive SEND in Kermit). ckcftp.c, 13 Sep 2000. Added FTP PUT /NOERROR which means to stop and fail if any upload fails; normally we just proceed to the next file. ckcftp.c, 13 Sep 2000. Change from Jeff for FTP setbpsz() (protection buffer size?) default value to improve performance of secure connections by cutting down on page faults. ckcftp.c, 13 Sep 2000. Went back and looked at Kermit Y code for ignoring gnfile() failures and don't see how it could have worked, so changed it to explicitly handle >0, 0, and <0 cases, and improved the transaction log record in this case. Tested with FTP and it works fine there too. ckcpro.w, ckcfns.c, ckuusx.c, 14 Sep 2000. Totally decoupled FTP TYPE from Kermit FILE TYPE, removed all references to the 'binary' variable from the FTP module, removed redundant calls to changetype(), and made PUT /BINARY and /TEXT work by saving ftp_typ and setting it as desired, and then restoring it upon the next entry to the FTP module (whose only entry points from outside are the FTP command and SHOW FTP). ckcftp.c, 14 Sep 2000. Fixed fpformat() to do something useful when both places and fp_digits are 0. ckuus4.c, 14 Sep 2000. But fp_digits should never be 0. However, it turns out that initfloat() was never called if FNFLOAT was not defined. Changed this to CKFLOAT. ckcmai.c, 14 Sep 2000. Added FTP PUT /PERMISSIONS, only for UNIX-to-UNIX; copies low-order 9 bits of each file's permissions to server copy after upload. ckcftp.c, 14 Sep 2000. Changed SEXP reader to ignore spaces between otherwise adjacent left parens (again). ckuus3.c, 14 Sep 2000. Moved GSOCKNAME_T SOCKOPT_T definitions from ckcnet.c to the end of ckcnet.h, and changed the FTP module to use them. ckcnet.[ch], ckcftp.c, 15 Sep 2000. Fixed a typo I made in dotlog() yesterday (":" -> ':'). ckuusx.c, 15 Sep 2000. Some AIX compiler warning fixes from Jeff: ckctel.[ch], ckcftp.c, 15 Sep 2000. Added dozens of (CHAR *) vs (char *) casts to ckcftp.c to eliminate prolonged howls of anguish and eventual death of the HP-UX ANSI compiler. It doesn't seem worthwhile to use CHAR in this module since we have to cast all the CHAR items to char anyway in order to use most APIs (but I didn't change the declarations, just put in casts as needed). Also initialized a bunch of variables that the compiler spuriously claimed could be used before a value was assigned. Now ckcftp.c builds without warnings (except "optimizations skipped") on HP-UX 10.20 too. 15 Sep 2000. Similarly for ckctel.c, 15 Sep 2000. Now also builds cleanly on Solaris 2.5, Tru64 4.0E, Unixware 7.1.1, RH Linux 6.1, and SCO 5.0.5, and not so cleanly on IRIX 6.5.8f -- thousands of warnings about variables defined but not referenced, set but not used. Usually I ignore these but I went through them this time and tried to handle all of them that were for variables with names longer than one char, and found some of the warnings were actually quite useful; e.g. rc (return code) being carefully set all over the place in some routine, but then not used in the return() statement. The SGI linker also picked up some cases of type mismatches between extern declarations and the original ones Also removed quite a few big buffers that were no longer used. Many files, 15 Sep 2000. Added several more CHAR/char casts to ckcftp.c. 18 Sep 2000. Removed #include and from ckcftp.c, since they prevent successful compilation on some platforms (e.g. Motorola 68K) and all this stuff should already be handled adequately in ckcdeb.h. 18 Sep 2000. Dat found a problem with SEXPs, in which the end of an operand that was a quoted SEXP was not marked, e.g. (foo '(a b) '(c d)), resulting in all but the last operand returning too much of the original string. Fixed in cksplit(): ckclib.c, 18 Sep 2000. Dat noticed that case-insensitive recognition of SEXP keywords was broken again -- another casualty of optimization rampage. Fixed in xlookup() and mxlook(). ckuus5.c, 18 Sep 2000. If "}" (without the quotes) by itself was given as a C-Kermit command, it would go off into some weird parsing never-never land. It's a strange special case, fixed with a strange special-case check. ckuus5.c, 18 Sep 2000. Jeff made some changes that increase FTP PUT performance by a factor of 12 (28Kbps to 337Kbps on a given connection). I tuned this a bit (replacing per-character function calls with a macro), raising it to 515Kbps. But it's still about 5 times slower than binary. ckcftp.c, 19 Sep 2000. Removed trailing junk from an #undef directive in ckcdeb.h, 19 Sep 2000. Changed SET FTP FILENAMES to include an AUTO setting and made it the default, ditto for FTP PUT /FILENAMES. AUTO means LITERAL if client and server are like systems, CONVERTED otherwise. Tested on UNIX client with UNIX and VMS servers; works OK. Added some crude code to compare client and server system types so this will work on Windows as well as Unix, but who knows what system types are reported by Windows FTP servers. To fix, search for "myostype". The variable "alike" should be 0 if client and server are not alike, and 1 if they are. If we can set this reliably, we can probably take a lot of shortcuts & liberties. ckcftp.c, 19 Sep 2000. Added \v(ftp_server) that shows the server type and also added this info to SHOW FTP. ckuusr.h, ckuus4.c, 19 Sep 2000. The SEXP reader didn't like numbers that started with '.' or '+' because the xxfloat() macro front end for isfloat() didn't account for them. Fixed in ckuus3.c, 19 Sep 2000. Dat found another fascinating bug; another case of a macro with a certain name, when used as an SEXP operator, not working; when I tried changing its name, it worked fine. There was one more place where the operator index (x) was being compared with a built-in operator code, when in fact it was a macro table index. Fixed in dosexp(), ckuus3.c, 19 Sep 2000. Made FTP set \v(cps). ckcftp.c, 20 Sep 2000. Converted FTP sendrequest() to use Kermit file i/o APIs. Speed is about the same (and text is still 5 times slower than binary). Enabled FTP PUT /COMMAND, works fine; ditto for /FILTER: and /ARRAY:. This also makes the FTP-format transaction log work. ckcftp.c, 20 Sep 2000. Hooked the STATISTICS command into FTP. ckcftp.c, ckuus4.c, 20 Sep 2000. The text/binary discrepancy is no big deal, since it is observed only on localhost connections; regular FTP clients behave the same way. When network i/o is not the bottleneck, it's the difference between big block read/writes and per-character loops. On a real network connection, the difference vanishes. Added some #ifdefs to STAT to allow linking on platforms where the new FTP code is not included. ckuus4.c, 21 Sep 2000. Dat noticed that (QUOTE (a b c)) and ('(a b c)) gave different results when used as a macro argument in an SEXP. Fixed in ckuus3.c, 21 Sep 2000. Started work on integrating FTP module with Kermit file transfer display. Changed handling of 'what' variable to use bit testing/setting, rather than testing for equality, since now the W_FTP bit can be set at the same time as W_SEND or W_RECV. Exported ftp_host; added a \v(ftp_host) variable for it. ckcdeb.h, ckcker.h, ckcfn[s23].c, ckuusx.c, ckcftp.c, 21 Sep 2000. Filled in details for fullscreen file-transfer-display for PUT/MPUT only. BRIEF works too, didn't test the others yet. ckcftp.c, ckuusx.c, 21 Sep 2000. Minor repairs to file-transfer display (missing #ifdefs, etc). Also added display of remote system type to FTP fullscreen file-transfer display. ckuusx.c, 22 Sep 2000. Replaced all the ftp_vbm/oldverbose/quiet/q/oq/oquiet manipulations with a much simpler scheme: ftp_vbx is the global, sticky version. ftp_vbm is is restored automatically from ftp_vbx (or set to 0 if quiet != 0) at the beginning of each FTP command. ftp_vbx is changed only by SET FTP VERBOSE-MODE; ftp_vbm (the one which is actually used) can be changed as desired without explicit saving & restoring (except when it needs to be turned off and back on or v.v. within a single command). ckcftp.c, 22 Sep 2000. Debugged SERIAL and CRT transfer display formats for FTP. ckuusx.c, 22 Sep 2000. Added SET FTP PERMISSIONS { ON, OFF, AUTO }, and added OFF and ON args for FTP PUT /PERMISSIONS. ckcftp.c, 22 Sep 2000. Added quoting of grouping characters in SEXPs. ckclib.c, 22 Sep 2000. Added \v(sexpdepth). ckuusr.h, ckuus4.c, 23 Sep 2000. Added \fcmdstack(level,flags) to provide programmatic access to the call stack. ckuusr.h, ckuus4.c, 23 Sep 2000. Fixed a bug I introduced in cksplit() several days ago, in which group quoting as in ('(a b c)) could cause us to run past the end of the string. ckclib.c, 23 Sep 2000. Extensive testing of cksplit() with quoting revealed an inconsistency in results when quoting was used in a word or group versus between them. When between words, we simply move the word-begin pointer past the quote, so it doesn't show up in the result. But within a word the quote character was kept because we don't copy the result to yet another new buffer; we simply insert NULs and point to pieces of the string. So in this case we have to scoot the part of the string before the quote to the right by one position to cover the quote. ckclib.c, 23 Sep 2000. Changed \fstripb() to accept a grouping mask just like \fsplit() and \fword(). ckuus4.c, 23 Sep 2000. Fixed evala() to return its result through ckltoa(). This (a) eliminates another sprintf(), and (b) removes the need to copy every evala() or evalx() result immediately to prevent it from being overwritten by subsequent calls; eval[ax]() returned its result from a single static buffer, whereas ckltoa() returns its result from a big circular buffer. Cost: 0. Benefit: Big (if there were any places in the code that called evala() or evalx() more than once in succession without copying the result each time, results would be wrong). ckuus4.c, 23 Sep 2000. The evala() change affects all \function()s that have more than one numeric argument. Went through all of these and removed all the free/malloc/copy code after evalx() calls. ckuus4.c, 23 Sep 2000. Added IF DECLARED to test if an array is declared. ckuus[26].c, 23 Sep 2000. Added HELP FUNCTION text for \fsexp() and \fcmdstack(); updated text for \fstripb(). ckuus2.c, 23 Sep 2000. Corrections from Jeff to BROWSER vs FTP #ifdefs, plus a couple FTP verbose-mode oversights. ckuus[r3].c, ckcftp.c, 24 Sep 2000. The following construction: define xxx { if > \findex({\(},{ab\(cd}) 0 { echo one echo two } echo three } confused the macro reader, causing the "echo three" command to be skipped. Adjusted the macro reader to pay attention to quoting of parentheses and braces. getncm(): ckuus5.c, 24 Sep 2000. Began work on FTP charset translation. Read through RFC2640 and was not impressed. The FEAT mechanism is silly. You can send pathnames and files in UTF8 or any other charset with or without it. The FTP partners are supposed to "autodetect" UTF8, which is nuts. Most of RFC2640 is devoted to the useless LANG feature (for messages). It barely even mentions data transfer. There is no notion of character-set tagging of a particular file or message. All that can be done is the server can indicate to the client that it (the server) supports UTF8. The client can't tell the server, OK, here comes a file in UTF8. It can't even say "Terrific, I support UTF8 too". Anyway, no known servers support the FEAT mechanism anyway. So in practice there is virtually no difference between a client that supports RFC2640 and one that doesn't. To Kermit I added: SET FTP CHARSET-TRANSLATION { ON, OFF } <-- Default OFF SET FTP SERVER-CHARSET <-- Default is UTF8 FTP PUT /SERVER-CHARSET:xxx /LOCAL-CHARSET:xxx <-- FCS values (GET will come later.) The local text-file character set is determined as in Kermit transfers. As always, the switches override the global settings & defaults. So far, parsing and SHOW FTP only; will fill in the actual translation shortly. The main task will be to get the "default defaults" right. In the absence of FEAT UTF8, there is no way of guessing what charsets the server likes. ckuusr.h, ckcftp.c, 24 Sep 2000. The PTY command only recognized "" for argument quoting, and didn't even do that right -- once an open quote was detected, the rest of the command line was grouped, regardless of any closing quotes. Replaced exec_cmd() with a new version that simply calls cksplit(), which not only fixes the problem, but also allows quoting with single quotes and braces. Also changed cksplit() to add a NULL element at the end of the split array (without affecting the count) so it can be passed to execvp(). ckupty.c, 25 Sep 2000. Yesterday's macro-reader change broke macros that contained full-line comments. Went back to 7.0 sources and saw that I had done all this before and then yanked it because of all the things it broke. But rather than roll back again, I tried another approach. getncm(): ckuus5.c, 25 Sep 2000. Back to FTP charset translation. Filled in setting of charsets from scanfile() plus ASSOCIATE tables. Filled in Unicode-based translation code for PUT. Updated file-transfer display to show FTP character sets rather than Kermit ones. ckcftp.c, ckcfns.c, ckuusx.c, ckc71.txt, 25 Sep 2000. Changed CHARSET to CHARACTER-SET in FTP commands for uniformity with other commands, even though it's really verbose... ckcftp.c, 26 Sep 2000. Converted some more old #ifdefs to NEWFTP to fix broken non-TCPSOCKET builds. Ensured successful building with -DNONET, -UTCPSOCKET, -DNOFTP, -DSYSFTP, and default NEWFTP. ckuus[r45].c, ckcdeb.h, 26 Sep 2000. Fixed SYSFTP build to substitute "ftp" if no FTP client has been set. ckuusr.c, 26 Sep 2000. Added CHECK FTP (succeeds only for built-in FTP). ckuus3.c, 26 Sep 2000. Broke FTP throughput stats in yesterday's charset changes. Fixed in sendrequest(): ckcftp.c, 26 Sep 2000. Added charset and file size info to FTP transaction log entries. Fixed per-file elapsed time and cps stats and brief-format transaction log entries ckcftp.c, ckuusx.c, 26 Sep 2000. Filled in ftp_reset() so FTP RESET command would work (it's supposed to log out the user without dropping the connection by sending REIN). The watsun server doesn't support this. ckcftp.c, 26 Sep 2000. Added \v(ftp_connected) and \v(ftp_loggedin). These are needed because "ftp open host /user:blah /password:xxx" might succeed in making the connection, but still fail to log in. In this case, it returns success, so scripts need a way to test whether login also succeeded. ckuusr.h, ckuus4.c, ckcftp.c, 26 Sep 2000. Changed all FTP commands (except OPEN) to give an error message and fail gracefully after parsing if there is no FTP connection. ckcftp.c, 26 Sep 2000. Changed /NOERRORS to /ERROR-ACTION:{PROCEED,QUIT} (less confusing), and added SET FTP ERROR-ACTION to let users set this globally. ckcftp.c, 26 Sep 2000. Added Kermit-style file (X) and group (Z) cancellation, including file-xfer display messages, transaction log entries, etc. ckcftp.c, 26 Sep 2000. From Jeff: SET TCP IGNORE-SHUTDOWN to work around Shiva bug. ckuusr.[ch], ckuus3.c, ckcnet.c, 27 Sep 2000. Dat discovered that local arrays didn't work any more (i.e. they had lost their localness). This happened when I moved common code out of dolocal() to vardef() several weeks ago; I lost the the pusharray() call. Changed vardef() to return array info, and dolocal() to check the array info and if the variable was an array, to call pusharray(). ckuus5.c, 27 Sep 2000. There's a problem with array links that nobody noticed yet: define tryme { local \&e[] array link \&e[] \%1 void \fsplit(\%2,&e) show array &e } declare \&a[1] ; This must be declared. tryme &a {here are some words} ; Call macro to fill \&a[]. show array a ; No change in \&a[]. \fsplit() calls dclarray() on \&e[], which is a link to \&a[]. What should dclarray() do in this case? Presently it does nothing useful. Fixed it affect the linked-to array. Seems to do the trick. ckuus5.c, 27 Sep 2000. While I was at it, I improved DECLARE array-name parsing to (a) detect syntax errors right away and report the error, and (b) if the item to be declared is not an array name, then if it's a variable, expand it and check again. So now it's possible to declare an array whose name is in a variable, and to have a macro declare an array whose name is passed as an argument. ckuusr.c, 27 Sep 2000. Consolidated FTP PUT /ARRAY code. ckcftp.c, 27 Sep 2000. Peter E complained about HP-UX hardware flow control. Checked it with USR modem at 57600 bps. Dialed and uploaded a 2MB binary file with no errors or problems (RTS and CTS lights both stayed on constantly). Downloaded the same file, and this time the RTS and RxD flashed in perfect synchronization. Stress-tested this by typing Ctrl-S at the file-transfer screen, which halted the receiver. After 2-3 seconds the RTS went off solid, and so did the RxD light. After 5 or 10 secs I typed Ctrl-Q and the transfer resumed without a hitch -- no errors, no retries. I repeated this several times; everything worked perfectly. The only problem is that SHOW MODEM always says RTS is off when it is on. There are no obvious typos in ttgmdm(); added debugging statements to the HPUX section of ttgmdm() in ckutio.c and rebuilt. The debug log clearly shows that (a) ioctl(ttyfd,MCGETA,&x) succeeds, and that (b) it is not reporting RTS as on. Specifics: the ioctl() call does not return a failure code, and the returned data is: 0x1a0020 == 6400040 octal (110100000000000100000 binary). The definitions from are: /* Define the bits within the mflag field */ #define MDRS 00000000004 /* Data Rate Select */ #define MDTR 00000000040 /* Data Terminal Ready */ #define MRTS 00010000000 /* Request To Send */ #define MDSR 00002000000 /* Data Set Ready */ #define MDCD 00000400000 /* Data Carrier Detect */ #define MRI 00000000010 /* Ring Indicator */ #define MCTS 00004000000 /* Clear To Send */ So HPUX is reporting DTR, DCD, CTS, and DSR, period. I don't see anything wrong with Kermit. QED, te absolvo. Updated hpux80 target from Peter E (more optimization suppression). makefile, 28 Sep 2000. Added FTP PUT /UPDATE. Seems to work but didn't have time to test much. ckuusr.h, ckcftp.c, 28 Sep 2000. Finished PUT /UPDATE, mainly sorting out the messages, etc (e.g. getreply() was messing up the file-transfer display when MDTM returned a "File not found" response). ckcftp.c, 29 Sep 2000. From Jeff: Remove SET TCP IGNORE-SHUTDOWN. ckuusr.[ch],ckuus3.c, ckcnet.c, 1 Oct 2000. Changed FTP to always send STRU F, MODE S upon initial connection to server. ckcftp.c, 1 Oct 2000. Fixed FTP to skip debugging messages during file transfer if FTP DEBUG is ON. ckcftp.c, 1 Oct 2000. Added PUT /RESTART, but only for MODE STREAM, STRU FILE (which we now send upon initial connection), and then only for binary files or when client and server platforms are alike. If /RECOVER (a.k.a. /RESTART) is given, . Check for conflict with other switches; if OK: . If (scanfile() says binary) or (alike == 1) we can restart. If OK: . Get size of remote file: First send TYPE I, so SIZE result will be file size in bytes. Then send SIZE to get size. If error or remote file is not found, cancel recovery. If remote is larger than the local one, cancel recovery. . Set TYPE back to file's type. . If file is still a recovery candidate, compare timestamps; if local file is newer than remote file, cancel recovery. . If recovery is not canceled, and file sizes are equal, skip this file. . Otherwise, seek to , and if successful, send REST , and check result. If result OK, STOR the file. Alternatively just APPEnd it. (Used APPE because it's more portable.) All of this works OK, except the result is corrupt -- some kind of off-by-4 error; I didn't have time to debug it. Also, still need to handle details of file-transfer display (start bar at right point, handle "skipped" message,etc), logging, settings, warnings, etc etc. To be cont'd. ckcftp.c, 1 Oct 2000. The problem yesterday was secure_flush() again -- the actions for secure versus clear were backward. OK, so now /RESTART works. ckcftp.c, 2 Oct 2000. Fixed file-transfer display for restart so thermometer and percent-done begin at the restart point. ckcftp.c, 2 Oct 2000. In K95 only, file-transfer display said "Kermit Protocol" even during FTP transfers. ckuusx.c, 3 Oct 2000. FTP PUT filename conversion wasn't working; the conversion was done, but the original name, rather than the converted one, was sent to the server. Fixed in putfile(): ckcftp.c, 3 Oct 2000. FTP PUT /RECOVER file-group could sometimes leave an input file open, causing subsequent PUTs to fail. Fixed in sendrequest(): ckcftp.c, 3 Oct 2000. Straightened out inconsistent sendrequest() return codes. ckcftp.c, 3 Oct 2000. From Jeff: fix a couple "not all control paths return a value" warnings; none of the Socket Option code was ever executed because of the value of ttmdm; fixed a potential denial of service attack; SET TCP commands should not be executed by users of IKSD. ckcmai.c ckuus3.c ckuus5.c ckcnet.c ckcftp.c, 4 Oct 2000. Added askmore() to SHOW FTP. ckcftp.c, 4 Oct 2000. Added protocol info to STATISTICS. ckuus4.c, 4 Oct 2000. Straightened out and documented success and failure of FTP PUT. ckcftp.c, ckc71.txt, 4 Oct 2000. As an experiment, added SET GET-PUT-REMOTE { AUTO, FTP, KERMIT }. If AUTO and an FTP connection is active, PUT (and MPUT, SEND, MSEND, MOVE, etc) use FTP. Ditto for BYE. Didn't do anything to the REMOTE commands yet; let's see how this feels first. ckuusr.[ch], ckcftp.c, 4 Oct 2000. Continued with GET-PUT-REMOTE, enabling it for REMOTE and the appropriate R-command shortcuts (RCD, RPWD, RDIR, RDEL, etc). Also BINARY, TEXT, LOGIN, LOGOUT, CLOSE, and HANGUP. ckuus[r7].c, ckcftp.c, ckc71.txt, 5 Oct 2000. Moved the code that sends SYST, STRU F, MODE S from ftp_open() to ftp_login(), since most servers won't execute these commands if user is not logged in. Turns out I had to do this in two places, so extracted the code to new ftp_init(), which is now called from both ftp_login() and ftp_user(). ckcftp.c, 5 Oct 2000. Finishing touches on GET-PUT-REMOTE work -- added a few more commands to the list (like FINISH), plus error messages for R-commands that don't apply to FTP. It seems pretty reasonable and intuitive; let's see what users think. ckcftp.c, 6 Oct 2000. Added network directory lookup for FTP. It doesn't do anything special -- just hostname lookup with verification that network type is TCP/IP. ckcftp.c, 6 Oct 2000. Added switch-parsing for GET, but so far it's parse-only -- nothing actually happens -- and still mostly code stolen from PUT, which needs to be adapted to GETting. ckcftp.c, 6 Oct 2000. Fixed secure_read() to call secure_getc() rather than secure_getbyte(), which fails on non-secure connections. ckcftp.c, 9 Oct 2000. Changed ftp_init() to set TYPE I if client and server are alike -- although we don't need this for PUT, we do need it for GET. ckcftp.c, 9 Oct 2000. Got FTP GET to work for a single file in both text and binary mode. Added code for client to ask server for the file's size so we can display % done, plus other hooks into file-transfer display. ckcftp.c, ckuusx.c, 9 Oct 2000. Added X/Z cancellation to FTP GET and made ^C work. ckcftp.c, 9 Oct 2000. Added transaction log entries for FTP GET. ckcftp.c, 9 Oct 2000. Added MGET, including tricky parsing for a list of filespecs, since we have to send an NLST command for each one separately. ckcftp.c, 9 Oct 2000. Filled in some of the easy switch actions: [ok] /TEXT (/ASCII) and /BINARY [ok] /AS-NAME (single file) [ok] /QUIET [ok] /DELETE (send DELE after successful reception) [ok] /ERROR-ACTION [ok] /EXCEPT (case sensitive if server is UNIX) [ok] /LARGER, /SMALLER (send SIZE for each file and compare) Both Jeff and Dat reported strange problems with scripts. In Jeff's case, the problem occurred in the host.ksc script because the GETMENUITEM macro definition was truncated. I traced this to one of my micro-optimizations in cmtxt() (avoid calling strlen() by referring to a variable that was supposed to already contain the length but apparently in some cases does not). Backing off on the optimization fixes Jeff's case and I hope Dat's too (no, of course not). ckucmd.c, 10 Oct 2000. Back to FTP GET. To do /AFTER, /BEFORE, etc, we need to be able to compare a remote file's date with a given date-time. But is the given date-time local, or expressed in the timezone of the server? We don't know the timezone of the server, and it doesn't make sense to use local time either, since that would be meaningless to the server. We could use GMT, but that doesn't match the client's OR the server's time (unless they are on the Greenwich Meridian). OK, one less thing to do. But for /UPDATE, we can do the same as for PUT, but backwards: if the local file exists, compare its GMT date with the remote's GMT date, and download only if remote is newer. ckcftp.c, 10 Oct 2000. Added as-name templates for FTP PUT and MGET. ckcftp.c, 10 Oct 2000. Fixed FTP {GET,PUT} /FILENAMES: and {SET,SHOW} FTP FILENAMES keywords and docs to agree. ckcftp.c, 11 Oct 2000. Added filename conversion for FTP GET. If as-name not given, and filenames are converted, apply zrtol(). ckcftp.c, 11 Oct 2000. Added filename collision options for FTP GET. Same as for Kermit; they follow the FILE COLLISION setting. ckcftp.c, 11 Oct 2000. Added /COLLISION: switch to FTP GET to override global FILE COLLISION setting. ckuusr.h, ckcftp.c, 11 Oct 2000. Added SET FTP COLLISION to allow a separate default collision action to be set for FTP. ckcftp.c, 11 Oct 2000. A couple typo corrections from Jeff. ckcftp.c, 12 Oct 2000. Debugged the FTP file collision code and got all the options working. ckcftp.c, 12 Oct 2000. Added FTP GET /NODOTFILE, /NOBACKUP, /RENAME-TO:xxx. ckcftp.c, 12 Oct 2000. Removed runique variable and related code from FTP module since we now handle filename collisions the Kermit way. ckcftp.c, 13 Oct 2000. Added MDELETE. It shares all the same code with MGET, but a smaller switch table. This is nice because we get all the same file-selection options, etc, and even a fullscreen display. ckcftp.c, 13 Oct 2000. Finished FTP MDELETE: file-transfer display details, transaction log entries, failure handling; merged FTP DELETE and MDELETE, since there is no reason to separate them, and linked RDEL to FTP DELETE. ckcftp.c, 14 Oct 2000. Added MGET /LISTFILE. ckcftp.c, 14 Oct 2000. Simplified file-transfer display management by hiding display initialization in the display routine itself, so initialization happens automatically the first time display is called for, but doesn't happen at all if no display is ever required, e.g. for "rdelete nonexistent file", in which case we print a simple error message. ckcftp.c, 15 Oct 2000. Fixed FTP display to put the server's error message in the Message line of the file transfer display, so commands like "get *.*" will give a meaningful indication of why they didn't work. ckcftp.c 15 Oct 2000. Discovered that { [M]GET, [M]PUT, [M]DEL } switches were sticky. That's because ftreset() was not being called after FTP transfers (as it is after Kermit transfers). Fixed by installing a call to ftreset() in the exit sequences of doftpget() and doftpput(). ckcftp.c, 15 Oct 2000. Added GET /RECOVER. Must send REST to server, so download recovery depends on this nonstandard feature. Otherwise it's like PUT /RECOVER: we require type binary or else alike > 0 and no Unicode charsets, then we compare sizes, dates, and so on, before attempting recovery. It works fine except a single NUL is always inserted at the recovery point. Why? It's not because of recovery; it's because interrupting a download with 'X' causes a NUL to be inserted before the final record. For example, when a download is interrupted with 'X': partial file size: 679936 (a multiple of 8192, the write-buffer size) NUL inserted at: 671745 (1st byte of final block) How could this happen? In binary mode, recvrequest() calls secure_read() in a loop, which in turn calls secure_getc(), which in turn calls iscanceled(), which checks for keyboard interruption. And sure enough, the trouble is that secure_read() deposits the secure_getc() result in the output buffer without checking first for cancellation. ckcftp.c, 15 Oct 2000. Got GET /RECOVER working in text mode, except it doesn't really work -- a piece is missing from the result at the restart point. The problem appears to be that REST, when given in "ascii" mode, is "interpreted" by the server (like it does for SIZE). However, you can't get around this by sending TYPE I, then REST, then TYPE A, then RETR, because nothing must come between the REST and the RETR. Therefore it is not possible to REGET in text mode. Poor old dumb FTP... To avoid the unnecessary delays, fixed iscanceled() to return 1 immediately if cancelfile is already set. ckcftp.c, 15 Oct 2000. Fixed FTP recvrequest() to return and/or set meaningful error indications rather than just fprintf(stderr)'ing them. ckcftp.c, 16 Oct 2000. Changed FTP recvrequest() to use Kermit API (zchko()) for output file-access checking. This gets us SET ROOT enforcement for FTP GET and also lets us remove bunches of ugly code from recvrequest(). ckcftp.c, 16 Oct 2000. Changed UNIX zchko() to set errno to EACCES on SET ROOT violation so perror(), ck_errstr(), etc, will give reasonable results. ckufio.c, 16 Oct 2000. FTP DIR didn't work if current directory was not writeable. Due to messed-up #ifdefs, recvrequest() always tried to create a file called "-". Fixed in ckcftp.c, 16 Oct 2000. Converted recvrequest() to use all Kermit APIs for file output, crudely, but apparently without breaking anything (e.g. recovery). I imagine it could stand some optimizing, but it's pretty fast already. Also cleaned up all kinds of scary code in the same routine that used uninitialized variables. ckcftp.c, 16 Oct 2000. Implemented FTP GET /COMMAND. Probably needs some refinement, e.g. regarding GET vs MGET. Didn't do /FILTER yet. ckcftp.c, 16 Oct 2000. Fixed saving and restoring of SEND and RECEIVE filters for FTP as well as Kermit PUTs and GETs. ckuus[r67x].c, 17 Oct 2000. Implemented FTP GET /FILTER. Still needs a lot of checking. ckcftp.c, 17 Oct 2000. Chased down a problem seemingly confined to highly complex and convoluted scripts. To do this I had to add a previously unused debugging format, F010, which is like F011, but treats s2 as a NUL-terminated string, and then replaced debug(F11x) calls that show command buffers, macros definitions, etc, with the new format to make the logs more manageable. By not showing (e.g.) an entire macro definition over and over and over in the log, I was able to reduce the size of script-oriented debug logs by a factor of ten. ckuus[56x].c, ckucmd.c, 18 Oct 2000. Analysis of the new trim logs points the finger at the following: While looking at RETURN, discovered that it set success=1 when given a value, and success=0 when not given a value, which makes no sense. Changed it not to mess with success. ckuus6.c, 28 Jun 99. This was a mistake. If a RETURN command is executed, it should cause the macro to succeed, period. Fixed in doreturn(), ckuus6.c, 18 Oct 2000. This still didn't fix the script in question. The reason is indeed subtle. Exiting from a macro by reaching the end (as opposed to an explicit END, STOP, or RETURN statement) does not set SUCCESS/FAILURE, by design. In this case, it returns the the status of the command most recently executed. Now recall that certain commands do not set SUCCESS/FAILURE, for example a failing IF command, an ECHO command, etc. The script in question was executing a macro which (a) executed a command that failed; (b) did not execute any subsequent commands that set SUCCESS/FAILURE; and (c) returned by running out of commands, rather than through an explicit END or RETURN. Therefore the failure status was returned by macro. However, if a "void" command was added to the macro, it returned success rather than failure. Of course, because VOID sets SUCCESS; the same would have occurred with any other command that succeeds, including SUCCEED. Moral: If you want to test a macro for success or failure, always leave the macro with an END statement, or some other statement that sets SUCCESS/FAILURE appropriately. Corollary: if a macro has an empty definition, or contains no statements that set SUCCESS/FAILURE, then invoking the macro does not change the SUCCESS/FAILURE status. Incidentally, it would be possible to simply initialize every macro's return status to SUCCESS, but that would rob us of the ability to write macros that test the status of the command that preceded the macro call. Jeff noticed that \fsplit() doesn't treat backslash as a separator any more. This happened during SEXP work, when it became necessary to allow for quoting of parens, etc, to override their separation or grouping functions. But this made it impossible to use backslash as a separator, no matter how many of them you used, and we want this to work, e.g. for DOS pathnames. So I added a special case to cksplit() for "\\" -- if the quoted character is itself a backslash, don't bypass the later test for whether it's a separator. ckclib.c, 20 Oct 2000. A user noticed that the statement: minput 20 {\13\10$ } {\13\10%\32} {\10\13$\32} {\13\10)\32} in our standard LOGIN.KSC script had stopped working because of the right parenthesis in the last pattern (used for AOS/VS). It's not really a bug; since the SEXP work, we can't be quite so free-and-easy with syntax -- any brace or parenthesis that is to be taken literally should now be quoted. Still, it's not good that this breaks an existing script for no good reason; an unmatched right paren should be ignored. Why isn't it? Added SET DEBUG LINELENGTH (invisible) so we can have short debug strings by default, but get long ones if we need them. This affects all debug(F010,...) statements, which I have attempted to use wherever a very long string might be entered into the log. While I was at it, I fixed the nutty flushleft switch() formatting of doprm(). ckuus3.c, 21 Oct 2000. Back to the LOGIN.KSC problem... Why does the unmatched right paren cause early termination of the macro? Because litcmd() replaces unmatched right parens with "\{41}" (to prevent early termination of the \fliteral() construction that is generated for, e.g. THEN and ELSE parts of IF statements to prevent premature evaluation of variables). This was fine before, but now that we recognize "\{" as a quoted left brace, the terminating brace matches some earlier opening brace rather than the immediately preceding one. This was fixed in litcmd() by making it generate "\041" rather than "\{41}", which is safe, because the \nnn interpreter never reads more than 3 digits. ckuus5.c, 21 Oct 2000. But now I wonder if this same problem might not occur in any script that uses \{nnn} notation. Talk about quoting hell... Let's try it: def foo { echo [\{65}\{66}\{67}\{68}\{69}] if success { echo [\{73}\{83}\{32}\{79}\{75}] } else { echo no good } } do foo Seems to be fine but let's keep our eyes open. Back to FTP... Realized that I did FTP GET /MOVE-TO and /RENAME-TO wrong. I have them affecting the source file on the server, but for consistency with the corresponding Kermit commands, they should affect the received file after it is successfully downloaded. ckcftp.c, 21 Oct 2000. Fixed parsing of "!command" when NOPUSH is set. ckuus5.c, 22 Oct 2000. Added redirection to FTP DIRECTORY, so we can send the server's directory listing to a file, a pipe, etc. ckcftp.c, 22 Oct 2000. Adding charset translation to FTP GET... We need to use xgnbyte(), which translates from any charset to UCS-2. However, it always uses zminchar() to get its next character. Added a function-pointer argument to tell it what routine to call to get its next input character, and changed all prototypes and invocations accordingly. If the pointer is NULL, we call zminchar() (which is a macro), as before. ckcker.h, ckcfns.c, ckuus[46].c, ckcftp.c, 22 Oct 2000. Added code to recvrequest() to translate charsets if in text mode. Added translation on/off plus file and server charset numbers as parameters to recvrequest() and getfile(), and to prototypes and all invocations of them. ckcftp.c, 22 Oct 2000. Started looking into GET /RECURSIVE. "NLST *" sent to WU-FTPD results in a recursive listing. We do a GET for each item but it doesn't work if the directories don't already exist on the client. As long as we're going between UNIX and UNIX, all I have to do is call zmkdir() on each name and recursion works. Then the question becomes: how to disable it? What does the regular FTP client do? The same thing: it receives the recursive list and tries to GET each file, and then fails for each one that contains a pathname if the directories in it don't exist. If they do exist, the recursive transfer works. What if the server isn't UNIX? If it sends pathnames in DOS or VMS format, we won't even know they are pathnames, and so won't call zmkdir() them, and will simply use the foreign pathname as the filename. This behavior is familiar to FTP users. OK, so GET /RECURSIVE means to check each name in the NLST file list for slashes or whatever. If a name has a directory separator, we try to create the directory, whereas if /RECURSIVE is not given and a name contains a directory separator, we simply don't ask for it. This is gross but it works just fine with WU-FTPD. ckcftp.c, 22 Oct 2000. Also PUT /RECURSIVE was broken, fixed now. ckcftp.c, 22 Oct 2000. At this point, all the major FTP features that I was going to do are done; all that remains is testing, debugging, porting, tuning, and cosmetics. Jeff needs to make sure the security features still work, and either fill in and debug the proxy stuff or else remove it, and maybe make HTTP more symmetrical. ckcftp.c has grown from 825 lines on July 15th to 10,675(!) -- hmmm, why's it so big? The object module is pretty hefty too: $ size ckcf*.o text data bss dec hex 33808 528 0 34336 8620 ckcfn2.o 27440 720 1560 29720 7418 ckcfn3.o 70712 1248 3184 75144 12588 ckcfns.o 75096 14840 26624 116560 1c750 ckcftp.o Probably because it combines user interface and i/o in the same module. Actually I suppose it's not that bad since /usr/ucb/ftp is about the same. Fixed f-t display "(SECURE)" indication for FTP. ckcftp.c, ckuusx.c, 23 Oct 2000. Added GET /TO-SCREEN, equivalent to Kermit REMOTE TYPE. We can't use /TYPE or FTP TYPE since they both mean something else (text vs binary). ckcftp.c, 23 Oct 2000. Changed UNIX nzltor() to accept a 3-state conversion argument, rather than the previous 2-state one. Previously, if the 3rd parameter was nonzero, full conversion was done. Now full conversion is done if it's > 0 and minimal conversion if it's < 0 (and none if it == 0, as before). Minimal conversion means everything except uppercasing of letters and enforcing only one period. Thus files like "Foo.tar.gz" can be sent without changing their names if we use a negative conversion argument, but we still do the other stuff like replacing space with '_'. The Windows and other versions of nzltor() should be changed this way too (but no harm is done if they aren't; they'll operate as before). ckufio.c, 23 Oct 2000. Changed FTP PUT to call nzltor() with a negative conversion argument if client and server are not alike but the FTP server is UNIX (I could do the same for Win32 FTP servers if I knew how they responded to SYST). ckcftp.c, 23 Oct 2000. Did the same for Kermit SEND (this time for both Win32 and UNIX servers). ckcfns.c, 23 Oct 2000. Added support for download directory. If someone goes to the trouble of setting one, FTP GET should use it just like regular GET does. ckcftp.c, 23 Oct 2000. Made the WHERE command work for FTP. ckcftp.c, 23 Oct 2000. Tested against IBM VM/CMS FTPD. Seems to work fine. Tested against VMS MultiNet FTPD. This one doesn't respond to SIZE or MDTM (OK...) but then it seems to somehow trick us into using the same name for the incoming file over and over. Which-buffer-is-which confusion in doftpget(), fixed now. ckcftp.c, 23 Oct 2000. When receiving a file and we don't know the size, then at the end we do "Percent done: 100 ////...." out to the end. However, it seems we don't clear the line first: Percent Done: 1009////////////////////////////////////////////////// ^ Fixed in ckscreen(): ckuusx.c, 23 Oct 2000. Fixed unprotected reference to ftpissecure() in ckuusx.c, 24 Oct 2000. Somehow on Oct 22, I removed the cmtxt() call from dolocal(). This causes compiler warnings and core dumps. Now it's back. ckuus5.c, 24 Oct 2000. Previously, the UNDEFINE command let you type lots of variables but only undefined the first one. Also there was no provision for undefining a group of variables based on a pattern. So I changed it to (a) allow a list of items to be undefined; (b) allow a pattern-matching option; and (c) add a verbose option, to list the variables that were undefined. Works nicely except (a) sometimes /VERBOSE gets the wrong numbers for the summary, and (b) it doesn't seem to work for array elements. Will fix (and document) tomorrow. ckuus6.c, 24 Oct 2000. Fixed yesterday's Percent Done fix to not break the thermometer for normal transfers. ckuusx.c, 25 Oct 2000. Fixed a couple of compilation glitches noticed on Ultrix: cfe: Error: ckuusr.c, line 6931: 'cmdstats' undefined cfe: Warning: ckuusr.c, line 10102: Number of arguments doesn't agree with number in declaration: return(dodcl()); ckuusr.c, 25 Oct 2000. While I was at it, I cleaned up NOSPL builds. ckucmd.c, ckuus[25rxy].c, 25 Oct 2000. Noticed that "make linux" did not include built-in FTP client. The #ifdefs in ckcdeb.h were all wrong. Fixed them. ckcdeb.h, 25 Oct 2000. Linux/PC builds: Full: 1874K NOFLOAT: 1840K NOFTP: 1785K NONET: 1674K NODIAL: 1742K NOLOCAL: 1428K NOUNICODE: 1572K NOCSETS: 1489K NOSPL: 1575K NOHELP: 1619K NOIKSD: 1825K NOSHOW: 1806K NOXFER: 1404K Various adjustments were needed in the same modules. 25 Oct 2000. Spent most of the day checking a user's misbehaving script. It seems that using an \fexec() invocation as a macro argument does something terrible. Reworked a number of debug() statements to make it easier to track, but didn't have time to actually fix anything yet. ckuus[56rx].c, 26 Oct 2000. Yesterday's script exposed two problems. First, when a macro argument contains an \fexec() reference, the parser calls itself recursively and changes the macro index variable, which was not on the stack. Putting it on the stack fixed the problem. Second, we never accounted for the possibility that the macro called by fexec would change the macro table, thus rendering the index invalid by the time we use it (as an argument to dodo()), e.g.: def bad undef good, return "haha" define good echo \%1 do good \fexec(bad) Obviously this could be fixed by looking up the macro name again after parsing the args, but not without killing script performance, so I added Yet Another Global Flag, mtchanged (macro table changed), which is set by addmac() and delmac(). We set this to 0 before parsing macro args, and if it is nonzero afterwards, then we relookup the name. Also now we recover gracefully from the case above (the macro itself disappeared after we looked it up but before we tried to start it). ckuus[r5].c, 27 Oct 2000. Fixed the debug log to include each command (using the new length-limited F010 format) in a line that starts with CMD, followed by (P), (F), or (M), to indicate the command source (prompt, file, or macro), followed by the command itself, plus indicators of command-level changes, similar to what is shown by TRACE. So now "grep ^CMD" gets a command history showing command file and macro entry and exit and the commands themselves. The F010 format for the actual commands pretty much restricts each entry to fit within an 80-column line, but for heavy debugging we can use SET DEBUG LINELENGTH. ckuus5.c, ckucmd.c, 27 Oct 2000. OPEN !READ and OPEN !WRITE did not strip braces/quotes. Fixed in doopen(): ckuus6.c, 27 Oct 2000. Jeff says "All entries in the makefile that include KRB5 must now link to -lgssapi_krb5 and those referencing SRP must now link to -lkrypto if NEWFTP is on by default." Added this (without testing), 27 Oct 2000. Changed zrename() to not choose between (atomic) rename() and (vulnerable) link()/unlink() at compile time, but rather to call rename() (if available) and then if it fails (or is not available), to call link()/unlink(), and if that fails, to copy the file, then delete the original. ckufio.c, 27 Oct 2000. Fixed some compilation glitches (#ifdefs, casts, unguarded references to h_addr_list[]) in ckcftp.c, 27 Oct 2000. Built with FTP and tested OK on HP-UX 8.00, HP-UX 10.20, Unixware 7.0, BSDI 4.1, IRIX 6.5, Tru64 4.0E, Solaris 2.5.1, AIX 4.3, and (without FTP) on VMS 7.2-1. Minor adjustments required to ckuus[r35].c, ckuusr.h, ckcftp.c, ckcpro.w. 27 Oct 2000. Later Christian Mondrup built it OK on Ultrix 4.4 too. And Gerry Belanger on Motorola Sys V/68 and /88 (although it doesn't actually work on those -- the TCP stuff isn't right, it times out trying to connect). Discovered that FTP PUT didn't show an error in its file transfer display if the destination file could not be opened. Fixed in putfile(): ckcftp.c, 28 Oct 2000. Reformatted ckuusy.c to be normal by indenting flushleft case statements. 28 Oct 2000. Added '-9' command-line option for FTP host. Works with -M ("My user ID"). If FTP connection made from command line without -S (Stay), BYE exits Kermit with status code based on FTP transfers. ckuusy.c, 28 Oct 2000. Added FTP command-line personality, very simple, no options to speak of. ckcmai.c, ckuusy.c, ckcftp.c, 28 Oct 2000. Added command-line options to actually transfer files; thus an entire FTP session can be initiated from the command line with no interaction or scripting required (if you can make a secure connection or you don't mind putting your password on the command line). Upload only. ckuusy.c, ckcftp.c, 28 Oct 2000. Added downloading from command line, but it took a lot of fiddling to get the file transfer display working, extraneous messages suppressed, etc. ckuusy.c, ckcftp.c, 28 Oct 2000. Fixed GET statistics; the first file was always 0 sec, the second file had the time of the first, etc. Fixed ftp -q to really be quiet. Cleaned up the cmdlineget/put() routines. Fixed ftp -D to CD even if -p or -g not given. Installed FTP LOGIN as an invisible synonym for FTP USER, and uncovered FTP REGET. ckcftp.c, 29 Oct 2000. Any user can make C-Kermit dump core by starting with -A (IKSD) and then Ctrl-C'ing it. It appears the jmp buffer isn't set up yet. Jeff added some preventive measures. ckcmai.c, 30 Oct 2000. The SWITCH statement failed to strip braces from around its control variable, and also didn't handle control variables with empty or multiword values very well. I retooled the SWITCH command (and the _FORWARD command that it uses internally) to allow all combinations of control variable contents (empty, one word, multiple words) and encapsulation: switch \%a { ... } switch {\%a} { ... } switch (\%a) { ... } switch ( \%a ) { ... } switch { \%a } { ... } switch { (\%a) } { ... } ckuus[r6].c, 30 Oct 2000. Added lots of debug() statements to ftp_hookup() in hopes of tracking down the problems on FreeBSD 3.1, SV68, and SV88. ckcftp.c, 30 Oct 2000. Back to FTP command line. The -Y option (skip init file) didn't work because it has to be done in prescan() but prescan() doesn't do anything if the command-line personality is not Kermit. Added code to prescan() to take care of this. ckuus4.c, 30 Oct 2000. Moved doftparg() from ckuusy.c to ckcftp.c, since it will need access to the symbols, variables, etc. 30 Oct 2000. Added security-related command-line options from kftp, parsing only; actions need to be filled in. doftparg(): ckcftp.c, 30 Oct 2000. From Jeff: "In ckxlogin() I added a doexit() to prevent an infinite loop in IKSD caused by the ttfdflg flag. This flag appears to be set when IKSD starts which results in ttclos() not closing the socket when it is called. I don't think the ttfdflg flag should be set when we are using fd 0 in IKSD. There is no parent process for us to preserve the socket for. I'm not sure what the best way to modify this is, so I leave it to you." We'll leave it like this for now. ckuus7.c, 31 Oct 2000. Filled in some of the FTP help text. ckcftp.c, 31 Oct 2000. Looked into the FTP connection failure on FreeBSD 3.x. The connect() in ftp_hookup() fails with EACCESS. "man 2 connect" says EACCESS is returned when "write access to the named socket is denied" or "search permission is denied for a component of the path prefix". These are listed under the heading: "The following errors are specific to connecting names in the UNIX domain. These errors may not apply in future versions of the UNIX IPC domain" (whatever that means). Yet Kermit's TELNET command works fine on the same computer. Comparing ftp_hookup() with netopen() in ckcnet.c, the first difference that jumps out is that ftp_hookup() simply calls inet_addr() on the name, whereas netopen() has dozens of lines of #ifdefs based on symbols NOMHHOST, INADDRX, and INADDR_NONE, and then the whole HADDRLIST business. In ckcnet.c, there are 241 lines devoted to address resolution, whereas in ftp_hookup() has only 37. Maybe we should just adapt netopen() for FTP and use it instead ftp_hookup(). This will probably also fix the other nonstarters too: In SVR4/88 connect() gets errno 157, "no route to host"; in SVR3/68, the errno is 115, "connection timed out". Another clue, by the way, is that the connect() fails only if you give a hostname; connect() works fine when you give an IP address. This is true on all of the failing platforms (FreeBSD 3.1, 3.3, and 4.1 as well as SV/68 and SV/88). Fix from Jeff for the FreeBSD FTP problem (HADDRLIST stuff). This also did the trick for SV/[68]8. ckcftp.c, 1 Nov 2000. Jeff filled in the network side of the TELNET COMPORT option. ckctel.[ch], ckuus[r3].c, 1 Nov 2000. There was some kind of malloc/free problem in the ftp code that caused core dumps in SV/68, which I don't have access to, and couldn't reproduce anyplace else. However, I discovered that in the very latest FreeBSD release, the free() routine prints an error message when passed a bogus pointer, and used this to find the (or at least a) hole, and plugged it. ckcftp.c, 1 Nov 2000. Actually the problem in SV/68 was not malloc/free at all, but an #ifdef foulup. This was the first platform that we tried this code on in which "printf" was not defined as a macro, and the "#ifdef printf" in recvrequest()'s ASCII case didn't have the appropriate #else part. ckcftp.c, 2 Nov 2000. More Telnet COMPORT code from Jeff: ckctel.[ch] + various ckuus*.c modules. 2 Nov 2000. The various clients of secure_getc() were checking its return code in different ways, which apparently could result in long delays at the end of a connection, when it returned EOF but the client checked for < 0, on platforms where EOF is not defined to be -1. Fixed in ckcftp.c, 2 Nov 2000. Added connection-log entry code for FTP, and made SHOW CONNECTION also show any active FTP connection. ckcftp.c, ckuus[35].c, 2 Nov 2000. More #ifdef fiddling for COMPORT, more debugging for FTP, ckctel.[ch], ckcdeb.h, ckcftp.c, 3 Nov 2000. #ifdef'd out all shutdown() calls in ckcftp.c since they seem unnecessary (each shutdown() call is before a close()) and they cause long pauses on SV/68 R3. 4 Nov 2000. More Telnet COMPORT code from Jeff. ckctel.[ch], 5 Nov 2000. Adapted Jeff's Telnet COMPORT code to ckutio.c. 5 Nov 2000. Changed tnsettings() to differentiate between 7 and 8 data bits using our hokey convention for hardware parity, and added a convention for calling it with a "don't change parity" argument. ckctel.c, 5 Nov 2000. Added calls to tnsettings() from SET PARITY, SET STOP-BITS, and SET SERIAL, to allow for changes after SET HOST. ckuus3.c, 5 Nov 2000. More Telnet COMPORT code from Jeff. ckcdeb.h, ckuus[r34].c, ckctel.[ch], ckutio.c, 6 Nov 2000. Removed tailing blanks and fixed long lines in ckctel.[ch], and fixed the formatting of the TN_COMPORT code. 6 Nov 2000. I noticed that TN_COMPORT was not being included in the SunOS and Solaris builds, along with certain other features. I traced this to the "built-in makefile entries" in ckcdeb.h -- they were too far down. I moved them up to where they would have the intended effect. This might also explain the foulup with Solaris hardware flow control in 7.0. Fixing it actually took a fair amount of juggling. Rainy-day project: make sure that for every #ifdef BLAH in this file, there are no #define BLAH's further down. ckcdeb.h, 6 Nov 2000. Added a separate SET SPEED table for TN_COMPORT connections, and code to use it whenever the Telnet COMPORT option has been negotiated on the current connection. ckuus3.c, 6 Nov 2000. Corrections to TN_COMPORT from Jeff, ckuus3.c, 7 Nov 2000. Corrections to FTP temp file handling from Jeff, ckcftp.c, 7 Nov 2000. Added istncomport() to centralize criteria for testing if connection is a Telnet COMPORT one, and so modules that do not already include ckctel.h don't have to do so just to make this test. ckctel.c, ckutio.c, ckuus[34].c, 7 Nov 2000. Added TN_COMPORT code for SET FLOW. This allows the full range of flow control types to be selected when a Telnet COMPORT connection is active, even when they are not otherwise available. Also I changed setflow() to choose the SET FLOW /MODEM flow-control type for Telnet COMPORT connections. ckuus[3x].c, 7 Nov 2000. Changed FTP DELETE to not use the regular file-transfer display, but just list each filename with "OK" or "Failed". ckcftp.c, 7 Nov 2000. Added support for FTP URLs. It's only for when Kermit is invoked as "ftp", and only works for "ftp:" URLs. If the URL includes a path (i.e. filename): . If no username is given, we log in automatically as user anonymous. . GET is implied -- we get the file. So for example: ftp ftp://kermit.columbia.edu/kermit/READ.ME gets the Kermit FTP site read-me file anonymously. If a user ID is given but no password, the password is prompted for. If no path is given, we just connect to the host and wait for a command. To handle URLs, I added a urlparse() routine to ckuusy.c, and then use it to check any command-line hostname to see if it's a URL. If so, it sets pointers to the pieces and we treat them as if they had been given as separate command-line arguments. The result is about the same as what we already had for Telnet: "telnet:" URLs are recognized when the command-line personality is Telnet. However, the FTP URL recognizer doesn't incorporate any notion of chaining to other services, and I'm not sure if this would be useful; after all, we don't expect FTP clients to make Telnet connections. ckuusy.c, ckcftp.c, 7 Nov 2000. The format of a comment in ckcdeb.h (in the #else part of #ifdef TTSPDLIST) was blowing up the SunOS non-ANSI compiler. Moved the comment delimiter and all is well. ckcdeb.h, 7 Nov 2000. Also a few last-minute adjustments to make sure we build OK with -DNONET, -DNOFTP, and -DNOCMDL. ckcftp.c, ckuus3.c, 7 Nov 2000. The temp[] buffer in remote_files() needed to be static so the file could be deleted when done. ckcftp.c, 8 Nov 2000. doftpget() was activating update mode unconditionally if FILE COLLISION UPDATE was set, but it should only have done this for GET, not for DELETE. ckcftp.c, 8 Nov 2000. Opening a Telnet connection would close any open FTP connection. Fixed in setlin(): ckuus7.c, 8 Nov 2000. Added \v(ftp_getputremote) to allow the FTP GET-PUT-REMOTE setting to be saved and restored in scripts. ckuusr.h, ckuus4.c, 8 Nov 2000. Generalized FTP server OS-type detection. ckcftp.c, 8 Nov 2000. Filled in more HELP FTP text. ckcftp.c, 8 Nov 2000. From Jeff: . Fixed I_AM_XXX detection for K95, ckcmai.c. . Fixed ftp ftp://kermit.columbia.edu/kermit/ to perform a directory listing instead of a performing a recursive get. . Fixed a prototype error in ckcfns.c. 10 Nov 2000. Changed "ftp ftp://xxx" to suppress messages. ckcftp.c, 10 Nov 2000. SHOW CONNECTION didn't notice when a Telnet connection had been terminated by the remote; it only noticed if you give a CLOSE command. This is a new problem since 7.0. Added dologend() calls in the places where "closed by peer" is detected. cku{con,cns}.c, 10 Nov 2000. Changed dologshow() to test the ttchk() return code for > -1 rather than != -1, since it was missing the -2 case. ckuus3.c, 10 Nov 2000. SHOW CONNECTION was truncating the last digit of the time. Another case of using ckstrncpy() (which is intended only for copying WHOLE strings) rather than strncpy() (which must be used when we want to copy a PIECE of a string). ckuus3.c, plus added comments to ckclib.c, 10 Nov 2000. Commands for SRP security options added to FTP by Jeff. ckcftp.c, 12 Nov 2000. Filled in remaining help text. ckcftp.c, 12 Nov 2000. Fixed FTP to use the networks directory the same way Telnet and other services do: cycle thru matching entries until one is found that works; there's no reason why FTP should be different. ckcftp.c, ckuusy.c, 12 Nov 2000. Improved the heading for FTP DIRECTORY. ckcftp.c, 12 Nov 2000. Improved some file-transfer display status messages for when GETs fail or the file is refused. ckcftp.c, 12 Nov 2000. Discovered that \fexec(BLAH ...) (uppercase macro name) didn't work because of a 1-character typo. Fixed in mlook(). ckuus4.c, 13 Nov 2000. If the LINES or COLUMNS environment variable has a non-numeric definition, curses library calls dump core in HP-UX. Added code to screenc() to check for this case and to disable the fullscreen display when it is encountered. ckuusx.c, 15 Nov 2000. I wrote a Kermit script that reads a C header file to find all occurrences of #ifdef BLAH followed by #define BLAH later in the file. Sample result: (/usr/fdc/kermit/) C-Kermit>take define.ksc ckcker.h -------- 56. #ifdef WHATAMI 306. #define WHATAMI -------- 210. #ifdef CK_APC 224. #define CK_APC 227. #define CK_APC 230. #define CK_APC -------- 260. #ifdef CK_AUTODL 274. #define CK_AUTODL -------- ckcker.h Lines: 1257 Possible errors: 5 Time: 64 sec All of these turned out to be OK. The real rat's nest is ckcdeb.h with 5800+ lines and deeply convoluted #ifdefs. This one took 2700 seconds (45 min) and turned up some duplicated #ifedf blocks mainly involving BLAHx implies BLAHy for SunOS and Solaris, of each which I removed the later instance. ckcdeb.h, 15 Nov 2000. Added the bit of code that checks if we had CD'd to a temporary directory (e.g. download directory) and if so, CD's back to where we were before to ftreset() as a catch-all, in case of Ctrl-C, etc. ckuusx.c, 16 Nov 2000. In UNIX ttinl() where we do sanity checks on the packet fields, in most cases we only look at the low-order 7 bits of the control fields, but in one case (length field) I neglected to ignore the high-order bit, resulting in spurious rejection of the first packet if it has parity, since we have not yet called parchk() on it. ckutio.c, 16 Nov 2000. Added SET TRANSFER MESSAGE to set a text message to be displayed initially in the "Last Message:" field of the fullscreen display. ckuusr.h, ckcmai.c, ckuus[3x].c, 16 Nov 2000. The stats script that uses SEXPs has a problem when the dataset includes big and/or many numbers: the sum of the squares goes negative: (++ xsum2 (^ x 2) ysum2 (^ y 2)) Why? Because an integer is being added to an integer, so it's not being converted to FP. The problem was in the "++" routine; it needed to check the result, just like the "+" routine does, and switch to FP upon overflow. Ditto for "--". ckuus3.c, 17 Nov 2000. Allowed SEXPs to be entered on multiple lines without having to hyphenate, until the right paren is entered that matches the opening left paren. ckuusr.c, 18 Nov 2000. Fixed fullscreen f.t. display to clear to eol before printing anything in the Last Message: field, in case SET XFER MSG had put something longer there previously. Also made ckscreen() print the XFER MSG in non-fullscreen displays too. ckuusx.c, 18 Nov 2000. TYPE /TAIL didn't work in K95 because the code in the "if (tailing)" sections did not account for the buffers containing UCS2 and therefore NULs. This required reworking all the code in those sections, and adding a parallel length array to save the length of each string. ckuus6.c, 18 Nov 2000. Suppose you want to use MINPUT to look for a bunch of targets simultaneously but you don't know in advance how many targets there will be. Previously there was no way. But now thanks to cksplit() and \fjoin() we can use arrays or array segments for this. If you put \fjoin(&a) or \fjoin(&a[3:5]) or whatever in the MPUT target list, it expands the array reference into the appropriate number of separate targets. This required a total rewrite of MINPUT parsing: from cmfld() in a loop to cmtxt() with subsequent cksplit(). ckuusr.c, 18 Nov 2000. When sending a file in local mode, we displayed the "Transfer OK" message up when we sent the Z packet, which was premature. For example, there might be dozens of unack'd data packets still in window. I moved the xxscreen() call for this from clsif() to the Y state of ckcpro.w. 18 Nov 2000. Added code for the '-9' (ftp host) command-line option to allow for a port and added help text for this option. In this case the port number must be appended to the hostname, connected by a colon (:) as in -j, -J, etc. ckuusy.c, 19 Nov 2000. The FTP command-line personality had no allowance for specifying a TCP port name or number. Handling this was a bit complicated. First of all, command lines like "ftp host:21" are confusing because they look like URLs. In fact there is no way of distinguishing this from a URL without "special knowledge". So we let our URL routine declare it to be a URL and then compensate for this afterward (if the URL has only service and host fields and the service is not "ftp" then we rearrange the fields and say the service if FTP). But we also need to allow "ftp host port" (no colon) for compatibility with other FTP clients, so I added this too. ckuusy.c, ckcftp.c, 19 Nov 2000. The change that was made to nzltor() to allow minimal conversions when the name-conversion argument is negative also needed to be done for nzrtol(), with the corresponding change to the calls in the FTP module in cmdlinget() and in the GET command. ckufio.c, ckcftp.c, 19 Nov 2000. Added TYPE /TRANSPARENT to explicitly disable character-set translation in the TYPE command. ckuus[r2].c, 19 Nov 2000. Added CLEAR KEYBOARD-BUFFER to allow scripts to flush typeahead. ckuus[r2].c, 19 Nov 2000. Added HEAD and TAIL commands, which are simply aliases for TYPE with /HEAD and /TAIL switches, respectively. ckuusr.[ch], ckuus2.c, 19 Nov 2000. Fixed top-level ?-help to fit on a 24-80 screen again, at least in C-Kermit. ckucmd.c, 19 Nov 2000. Discovered that "if defined \v(blah) ..." gave an obnoxious error message if there was no such built-in variable, with no way to shut it up. Fixed by adding another special case to boolexp(): ckuus6.c, 19 Nov 2000. Added (invisible) DEBUG OFF/ON to give slightly more fine-grained control over debugging. If LOG DEBUG is active, this turns logging off and on. If LOG DEBUG is not active, or Kermit was built without debugging, it does nothing. Also converted a debug() straggler in dofor() to use F010 instead of F110. ckuusr.[ch], ckuus[6x].c, 22 Nov 2000. Suggested by Aaron Rendahl : unset IGNPAR bit when using hardware parity, so incoming bytes that have parity errors are not discarded. ckutio.c, 22 Nov 2000. I hadn't done anything fun for a while, so started adding learned scripts. These are included if CKLEARN is defined (and NOLEARN is not defined). Initially it's just for UNIX (ckucns builds only); it can be added for K95 and VMS (when it is, update the CKLEARN #ifdefs in ckcdeb.h). Definitions: ckcdeb.h, ckuusr.h. Parsing: ckuusr.c. When learning, Kermit records all commands except LEARN and CONNECT, and it converts TELNET to SET HOST: ckucmd.c, ckuusr.c. Ctrl-C cancels script recording. ckuusx.c. SHOW FEATURES, SHOW SCRIPTS: ckuus5.c. HELP: ckuus2.c. 23 Nov 2000. Added code to the UNIX CONNECT module to record keystrokes and communications input as OUTPUT and INPUT commands, respectively, and it sort of worked, except it got really ugly with remote echoing. This was easily fixed by making it line-oriented rather than byte-oriented, at the expense of being able to script things like pressing the space bar at a "More?" prompt or editing a file with EMACS. The alternative would be some kind of horrendous super-hairy and error-prone echo-cancellation technique that would probably never work in real life. Better to keep it simple. ckucns.c (NOT ckucon.c; it can't be done there because forks don't share variables), 23 Nov 2000. The new "minput n \fjoin(&a)" feature would sometimes include a spurious empty element in the list, which, of course, would always succeed, thus wrecking any script. Rewrote the argument-gathering code to ensure there could be no empties. ckuusr.c, 24 Nov 2000. A minor refinement to learned scripts: fixed the INPUT timeout interval after long-running commands (like SEND). Also there were a couple bits of new code that were not within #ifdef CKLEARN..#endif; I added the #ifdefs to make it easier to copy to the code to other platforms. ckucns.c, 24 Nov 2000. Added -DNOLEARN to most wermit-based makefile targets. Made sure it built OK with -DNOLEARN. makefile, 24 Nov 2000. Adapted learned-script code to VMS and cleared up a few other VMS compilation glitches, like missing prototypes for Telnet Com Port functions. ckcdeb.h, ckuusr.c, ckvcon.c, ckctel.h, ckuus3.c, 24 Nov 2000. Fixed learned scripts to always record recalled commands, rather than only when they were different from the previous command in the recall buffer. ckucmd.c, 25 Nov 2000. Removed a vestigial cmcfm() call from EXIT parsing (cmtxt() had already been called). ckuusr.c, 25 Nov 2000. Discovered that addcmd() was called for every command, even though it only had an effect for commands entered at the prompt -- a lot of useless overheard for script execution. Changed the calls to depend on the appropriate conditions. ckucmd.c, 25 Nov 2000. Noticed that floating-point ops occasionally printed "log10: SING error". Fixed in fpformat() by not trying to take the log of 0. ckuus4.c, 25 Nov 2000. Verified that MINPUT n \fjoin(&a) now works properly in latest K95 build. 27 Nov 2000. Fixed cmcfm() to call addcmd() only if cmflgs not 1 (in case confirmation was already obtained, e.g., in cmfdb()). ckucmd.c, 27 Nov 2000. Minor corrections from Jeff to learned-script, FTP, Telnet Com Port, and SSL code. ckcftp.c, ckuusr.c, ckctel.c, ck_ssl.c, 28 Nov 2000. Made ADD SEND-LIST work with FTP. ckcftp.c, 28 Nov 2000. Ctrl-C during ftp login at Password: prompt was caught but just returned to where it was, i.e. didn't interrupt. That's because ftpcmd() had already been called, which sets SIGINT to cmdcancel(), but cmdcancel() only sets a flag and returns. Meanwhile we're inside readpass() which doesn't know anything about the flag, so readpass() keeps trying to read the password. Fixed by calling sigint() before calling readpass() (4 or 5 places). This should be harmless since the next ftpcmd() call will send SIGINT back to cmdcancel(). ckcftp.c, 28 Nov 2000. FTP to VMS / MultiNet... Interruption of GET with X or Z or ^C didn't work -- just stopped it cold. But interrupting the same kind of download from a Unix server worked fine. Furthermore, when using K95 as a client to the same VMS server, cancellation also worked. Eventually I figured it out. In UNIX, BSDSELECT is not defined, so select() was never being called and were hanging in getreply(). So I defined BSDSELECT and poof, everything worked. ckcftp.c, 28 Nov 2000. When GET'ing files from an FTP server that does not understand the SIZE command, the file-transfer display inappropriately put up the "Percent done" label, instead of the "Bytes so far" label. Fixed in ckcftp.c, 28 Nov 2000. In enabling the BSDSELECT code in ckcftp.c yesterday, I opened the "time.h" hornet's nest. I had foolishly hoped to sidestep it by defining an ersatz timeval-like struct with a different name, but this stopped compilation cold on HP-UX, SV/68, etc. Jeff thought a (void *) cast would fix it, and maybe it would if enclosed in #ifdef CK_ANSIC..#endif (void is not portable), but this seemed too easy (experience teaches that the easy way never works in C) so instead I went back and did it "right" with all the #ifdefs for , , , etc etc, as in ckutio.c. But it was easier than expected -- I forgot I had cleaned up this mess considerably in ckutio.c a while back. Built OK on SunOS/gcc, Solaris with Sun CC, HP-UX 9.05 and 10.20 with bundled non-ANSI compiler, HP-UX 10.20 with ANSI optimizing compiler. It blew up on HP-UX 9.05 with the ANSI compiler: not the timeval argument, but on the fd_set ones, which the HP prototype says must be (int *) rather than (fd_set *). Added #ifdefs for this -- no doubt this process is to be repeated a hundred more times -- then it built and worked OK. ckcftp.c, 29 Nov 2000. From Jeff: Enable learned scripts in ckcdeb.h. Updated Kerberos builds in makefile. Forward-X updates to ckcnet.c. 29 Nov 2000. From Jeff, 30 Nov 2000: ckctel.c ck_ssl.c ckcnet.c ckcfn2.c ckutio.c makefile More fixes to Forward X Added support for embedded TLS compression (Zlib). I see a 200% file transfer improvement on my DSL line. Corrected a problem with ttinl() and streaming when telnet negotiations are sent during a streaming receive. If there is incoming data while we are reading Data packets we were treating it as an error on the link even if the data was not in fact a Kermit packet. Instead, what we want to do is read the data, if it is a packet return an error OR if all the data was read and there was no packet, then return 0 bytes read. If 0 bytes are returned from ttinl() while in streaming mode, treat it as if we never called ttinl() in the first place. I tested it on Unix and Windows with several different combinations of data flows. Fixed a compiler warning in Solaris about the 2nd arg to recv(). For some reason I believed that it had be be "const char *" on Solaris, but it didn't. I wonder why I thought that... ckcftp.c, 30 Nov 2000. Back to ckcftp.c and select() vs time.h... It still didn't build on SV/68; now we have multiple definitions of struct tm. Rearranged #ifdefs til it worked. ckcftp.c, 30 Nov 2000. Ever since IRIX 6.5 came out, building C-Kermit there has been a big pain since the compiler spits out thousands of "variable declared but not used" or "variable set but not referenced" messages. I finally discovered the magic incantation to suppress these messages (-woff 1174,1552). makefile, 30 Nov 2000. Built on AIX 4.3, which went OK except a few modules exceeded the optimizer. But "ftp 128.59.39.2" didn't work -- it didn't do anything at all, just gave another prompt. But "ftp =128.59.39.2" worked fine. Network directory lookups were not working. First: don't bother with directory lookup if hostname starts with a digit. Second, "tcp/ip" was required in the network-type field and "tcp" wasn't accepted. Plus certain cases were simply not handled. Fixed in ckcftp.c, 30 Nov 2000. Portability checks/builds. OK if it builds without complaint, makes an FTP connection, transfers files, and file transfer can be interrupted (which is where select() comes in): . Unixware 7.1 OK . IRIX 6.5 OK (and now with no warnings). . Compaq Tru64 4.0E OK. . HP-UX 8.00 non-ANSI OK. . HP-UX 8.00 ANSI (ran out of memory after 4+ hours) . HP-UX 9.05 still OK. . HP-UX 10.20 ANSI OK. . HP-UX 10.20 non-ANSI OK. . DG-UX 5.4R4.11 OK. . SCO OSR5.0.5 OK. . FreeBSD 2.2.8 OK. . FreeBSD 3.3 OK. . Linux Red Hat 6.1 OK. . Solaris 2.5.1 gcc OK. . Solaris 2.5.1 Sun cc OK. . Solaris 7 gcc OK (with some .h redefinition warnings: int8_t, int16_t, ...) . Solaris 7 Sun cc OK. The QNX 4.24 build blew up big-time. More BSDSELECT fallout. In QNX, fd_set is defined in select.h rather than types.h. Adding the appropriate #include made it compile and link OK, but at runtime it dumped core in the first ftpcmd() call; I couldn't find any good reason for this. Tried building it on QNX 4.24. Here it compiled OK, but the link step crashed with SIGSEGV, "make error 139" (no clue what 139 is, but it's not an errno, and the online 'make' and 'ld' manuals do not list error codes). Nevertheless, the executable was there and could be started. The debug log shows that the very act of calling secure_command() is what makes it crash. secure_command itself never even starts. I can't imagine any reason for this. I tried changing the name of the routine and its args in case any of them were reserved words, but no change. Then, noticing that secure_command() (now renamed to scommand()) has a lot of automatic arrays that are quite big. So I #ifdef'd out the contents of this routine and made a short simple one with no automatic arrays (and no security) for QNX. Now we can call the routine, and it builds and sends the FTP command and even returns, but (a) getreply() says it gets 421 (EOF), and (b) we never get another prompt back. Then I tried making a Telnet connection -- hmmm, that hung too. OK, let's back up and see where we went wrong. Rebuilt with -DNOFTP, still can't Telnet. Well, there's no reason Telnet should have stopped working; of course it worked in C-Kermit 7.0. But QNX is very different from other UNIXes -- the makefile entry (written by Dan Hildebrand before he died) has some cryptic compilation and link options that I don't understand that probably have to do with memory model, segments, pages, etc, so maybe it's a size thing. Tried building with -DNOFTP and -DNOCSETS to remove two of the largest modules. No difference. OK, the SIGSEGV and "Error 139" were red herrings. It turns out the makefile entry not only compiles and links Kermit, but also tries to install some kind of "use message" using a utility called "usemsg" and that's what was crashing. Removed that from the makefile and no more scary messages. But it doesn't help at runtime. "set telnet debug on" shows all the negotiations are fine; the hangup occurs only after entering CONNECT mode, indicating that select() never returns. But select() worked in 7.0? What's the difference? Who knows, it could be anything -- the many changes in ckc{tel,net}.[ch], some module or routine whose size grew beyond some limit... After finding Watcom C docs, discovered that the -ms option in the qnx32 target means "small model" (64K). Changed this to -mf ("flat" 4GB) and removed QNX hacks from ckcftp. Now: . FTP still dumps core with SIGSEGV when ftpcmd() calls scommand(). There is nothing illegal or strange about this simple call; it works on all the other platforms. So -ms/-mf had nothing to do with this. . When Telnet first calls select(), the result is -1 with errno = EBADF. This seems to indicate that the FD_SET() setup is wrong. But this code hasn't changed since 7.0 (** not true, keep reading **), which works fine. Again, -ms/-mf irrelevant. . Autotelnet? Maybe scripts work if we don't enter CONNECT mode. But in trying this I discovered that the TYPE command also seg-faults. So do DATE, SEND, IKSD, ... OK so what else do we find in the CFLAGS? "-3r" means use register-based argument passing. Changed this to "-3s" (stack-based) and FTP and TYPE commands still crash in the same way, but now so does TELNET. Various other options (-r, -sg, -zt) made no difference either. Finally I went to QSSL for help and was told about the undocumented -N linker flag, which fixed everything except the hanging Telnet. Put the other options back the way they were (turns out -ms only means 64K in the 16-bit world; in the 32-bit world it means: separate code and data segments each with a max size of 4GB). makefile, 1 Dec 2000. So why doesn't TELNET work? Comparison of ckucns.c from 7.0 and the current one shows that the 1st (width) arg to select() was increased from 16 to 256 (why?); putting it back to 16 fixes the problem (I did this in #ifdef QNX but I wouldn't be surprised if something like this crops up elsewhere). ckucns.c, 1 Dec 2000. When adding learned scripts, I used addcmd() (which adds interactive commands to the recall buffer) to record commands. But this had certain problems; for example, some commands would be skipped in the learned script. That was because addcmd() didn't record a command if it was the same as the previous one. Anyway, I juggled things a bit to make this work, but evidently broke command recall. Fixing it required some pretty deep thought (like I should have done to begin with). The calls to addcmd() have always been at the wrong level. There really should be just one call: in gtword(), when the user types CR. So I removed all the old calls and put the one call there. This works fine except for one thing: cmfdb() can cycle through the same command several times, each time using gtword(), and therefore addcmd() can be called multiple times per command. A good example is DIRECTORY. This was circumvented easily by Yet Another Global Flag (but static to the command module): newcmd is set to 1 by cmini() and tested by addcmd(). If it is 0, addcmd() just returns; if newcmd is nonzero, the command is added and newcmd is set to 0 so it won't be added next time. ckucmd.c, 1 Dec 2000. The large width arg to select() was for Forward-X. Redid the select() call using FD_SETSIZE, whose definition is taken from the header files, and if not found there is defined to 256 if CK_FORWARD_X is defined, otherwise to 32. ckucns.c, 2 Dec 2000. Gave SET TERM IDLE-SEND its own #ifdef feature symbol, CKTIDLE, and defined it by default for OS2 and for UNIX unless NOLEARN was defined (NOLEARN pretty much corresponds to the UNIX targets that use ckucon.c instead of ckucns.c, and is set in all the appropriate makefile targets). ckcdeb.h, ckuus7.c, 2 Dec 2000. Changed SET TERM IDLE-SEND parsing to not be OS2-specific. ckuusr.h, ckuus7.c, 2 Dec 2000. Enabled select()-based SET TERM IDLE-SEND in UNIX select() builds, including (because why not) allowance for a negative timeout value to indicate milliseconds. ckucns.c, 2 Dec 2000. Added SET TERM IDLE-SEND to SHOW TERM and HELP SET TERM. Also added SET TERM TRIGGER to these, which had been omitted outside K95 in 7.0. ckuus[27].c, 2 Dec 2000. Updated NEWS text. ckuus2.c, 3 Dec 2000. The select() call in ckucns.c needed special treatment for HP-UX 9.xx and earlier, where the interior args must be (int *) rather than (fd_set *). ckucns.c, 4 Dec 2000. The tn_comport code is a major pain with its long identifiers, some of them about 40 characters long, and some of them not unique within the first 30. 31 is the max in VAX C and I'm sure others have smaller maximums. It's silly to disable a feature that would work otherwise just because the identifiers are too long. To fix, I wrote a very dumb shell script that uses sed to shorten the names of 58 overlong identifiers and ran it on ckctel.[ch], ckutio.c, and ckuus4.c, which are the only UNIX modules where these identifiers appear and rebuilt the result. (You have to run it 2 or 3 times on some modules because it only makes one replacement per line.) 4 Dec 2000. Added Telnet Com Port Option to VMS. ckvtio.c, 4 Dec 2000. Correction from Jeff for Kermit protocol timer initialization. ckcfns.c, 5 Dec 2000. Removed a spurious second inclusion of from ckcftp.c. 5 Dec 2000. More #ifdef twiddling for building on many platforms, mostly to do with and structs timeval and tm. ckucns.c, ckcftp.c, makefile, 6 Dec 2000. Build problems: . VMS 7.2 + TGV 4.2A on Dopey: conflicts in VMS and TGV header files for functions we aren't even using prevent successful build. Ditto MVB. . SCO OSR5 - DNS_SRV, SOCKOPT_T: ckcnet, ckcftp (just warnings, net only). The warnings make no sense, since SOCKOPT_T is int for SCO, and arg 5 is int in sys/socket.h. Same deal for getsockname(), accept() Various touch-ups from Jeff, plus addition of \v(secure). ckuusr.[ch], ckuus[347].c, ckcfn2.c, ckctel.[ch], ck_ssl.c, 7 Dec 2000. From Mark Berryman, a fix for a problem introduced in combinations of recent DECC / MultiNet combinations, in which both declare select() in their header files, but with conflicting data types. ckuusr.c, ckvcon.c, 7 Dec 2000. sco32v4net won't build no-how no-way; all the CONST/const garbage in ckclib (e.g. ckmakmsg) blows it up: "lvalue specifies const object", "lvalue specifies const object", etc. Adding -DNOANSI and/or -U__STDC__ doesn't change anything. ---7.1.199 Alpha.01--- brstrip() (strip braces or doublequotes from around string) didn't work for empty {} or "". ckclib.c, 11 Dec 2000. Added -DHWPARITY for SunOS 4.1; verified to work by Piet Vloet at Siemens. makefile, 11 Dec 2000. There has never been a way to make C-Kermit not send I packets when given a GET, FIN, REMOTE, or BYE command. Now there is: SET SEND I-PACKETS OFF. Needed because some Kermit server implementations can't handle I-packets. (Theoretically this was handled already: C-Kermit ignored an E packet if it came in response to an I packet; but it seems some Kermit servers are so buggy they send a NAK rather than than E if they get an I packet, so the only recourse is not to send the I packet.) ckuus7.c, ckcfns.c, ckcmai.c, 12 Dec 2000. Peter E has a program that generates Kermit scripts that contain constructs like this: xif condition {,- ,- command,- } In such constructions, the command is never executed (this only happens when ",-" is used). Diagnosis: the isolated ",-" confused the brace counter; it backed up too far and recounted the opening brace, and therefore never found an end to the braced sequence. Fixed in getnct(): ckuus5.c, 13 Dec 2000. Noticed that FTP remote_files() ignored its first argument, the one that says whether to build a new list or just return the next item from the current one. Instead it always built the list if there was no list, and then returned the next item from it. Wouldn't this mess up if we interrupted a GET and started a new one? Yes. Not only that, but the second GET resulted in a steady stream of "socket: Too many open files". Fixed in remote_files(): ckcftp.c, 15 Dec 2000. Added FTP CHECK remote-filespec. Client does NLST for given filespec, and succeeds if any files match, fails if none do. This lets the client check if a server file (or file group) exists. ckcftp.c, 15 Dec 2000. Added FTP [M]GET /NAMELIST:[filename]. This is just like [M]GET, except instead of getting the files themselves, it gets a list of their names into the given file. All the other GET switches are honored, so this is the list of files that matches the GET filespec and all the other selection criteria. The file is suitable for processing by the filter or software of your choice, as well as by Kermit itself (fopen/fread/fwrite/fclose), and as /FILELIST: file. If the filename is omitted (or given as "-"), the list goes to the screen. ckcftp.c, 17 Dec 2000. FTP PUT /AS-NAME:\v(filename).tmp * didn't work, due to some typos, fixed now. ckcftp.c, 18 Dec 2000. Discovered that PUT's asname buffer (asnambuf[]) was not cleared between PUTs, so the as-name was persistent. Fixed in ckcftp.c, 18 Dec 2000. Added [M]PUT /SERVER-RENAME:. This was the final missing piece of "atomic" file movement when uploading to the FTP server. Normally to be used with \v(filename). Requires write and rename access on the server, so doesn't usually work (e.g.) with anonymous uploads to public incoming areas, where the permissions don't allow renaming. Example: ftp mput /server-rename:\v(filename).ok * to append ".ok" to the filename when it's finished uploading. Or it can be used in conjunction with /AS-NAME to give a temporary name while uploading is in progress and revert to its real name when uploading is complete: ftp mput /as-name:\v(filename).tmp /server-rename:\v(filename) * It can also be used to move the file from the working directory to a final directory when upload is done: ftp mput /as-name:\v(filename) /server-rename:../final/\v(filename) * but in this case you have to know the pathname syntax of the server. If the rename fails, the [M]PUT command fails according to the ERROR-ACTION selection. There is no explicit /SERVER-MOVE-TO: switch because it would only do the same thing as /SERVER-RENAME (i.e. send RNFR and RNTO), but require risky code to construct the full pathname, which would require Kermit (rather than the user) to know the pathname syntax on the server, which is a protocol no-no. ckuusr.h, ckcftp.c, 18 Dec 2000. Added [M]GET /SERVER-RENAME:, same deal as [M]PUT /SERVER-RENAME:. ckcftp.c, 18 Dec 2000. Discovered that TOPS-20 gives only a filename list if you send LIST: C-Kermit>ftp dir Directory of files at cu20b.arpa: 227 Entering Passive mode, use PORT 192,94,202,40,13,20 150 List started. PS: CKERMIT.DIRECTORY.1 CKERMIT2.TXT.1 GNUEMACS.INIT.1 KERMIT.DIRECTORY.1 LOGIN.CMD.1 226 Transfer completed. If you want a full listing with dates, permissions, etc, you have to send NLST and then send STAT for each filename that is returned, and then read and print each response on the control socket (not the data socket). Added FTP VDIRECTORY (which is a familiar TOPS-20 command) for this: C-Kermit>ftp vdirectory *.* Directory of files *.* at cu20b.arpa: PS:CKERMIT.DIRECTORY.1;P20200;A,0,15-Jan-1999 13:41:26-EST, 15-Jan-1999 13:41:27-EST,14-Sep-1999 17:49:06-PDT,FDC,FDC PS:CKERMIT2.TXT.1;P775200;A,166,15-Jan-1999 13:43:17-EST,15-Jan-1999 13:43:17-EST,15-Jan-1999 14:35:06-EST,FDC,FDC PS:GNUEMACS.INIT.1;P775200;A,8,25-Jan-1997 10:46:54-EST,25-Jan-1997 10:46:54-EST,25-Jan-1997 10:49:34-EST,FDC,FDC PS:KERMIT.DIRECTORY.1;P20200;A,0,25-Jan-1997 10:45:59-EST,25-Jan-1997 10:45:59-EST,14-Sep-1999 17:49:07-PDT,FDC,FDC PS:LOGIN.CMD.1;P775200;A,1,25-Jan-1997 10:43:05-EST,25-Jan-1997 10:43:05-EST, 8-Sep-2000 16:15:37-PDT,FDC,FDC ckcftp.c, 19 Dec 2000. We need a way to specify whether the rename in GET /RENAME should overwrite existing files and/or if it should check first and fail immediately if the rename target already exists. GET /RENAME: /NOSUPERSEDE? But /NOSUPERSEDE is confusing (which end?) and makes no sense without /RENAME:, so really we need a single switch that means "get and then rename the source but only if the rename target doesn't already exist", so /SERVER-XRENAME: or something, yuk. This would work by checking the rename target before downloading (if it's there fail), and again after downloading (just before the renaming). But how do we know if it failed to download at all, or downloaded but failed to rename? There's no good way, we fail the same way in both cases. Usually it will be because the file was already there before we started the GET. OK, so GET /SERVER-XRENAME:. But oops, to do this we have call remote_files(), which will clobber the current file list. Which means we'd need to push and pop lists, with all the associated bookkeeping and file opening/closing. Is it worth it? I don't think so. Anybody who is concerned with this level of safety is going to want detailed per-file control at the script level and so wouldn't use GET /SERVER-XRENAME for a wildcard group anyway. So for Alpha.02, we just need to document how to get the file list and loop through it, embedding the current 'upload' script in a loop. Added TYPE TENEX (and {GET/PUT} /TENEX) to the FTP client -- no big deal. It's the same as binary except we send TYPE L 8 instead of TYPE I; all the packing/unpacking work is done on the other end. The hard part isn't coding it, it's explaining it: TYPE TENEX is to be used for uploading 8-bit binaries to 36-bit platforms and downloading them again. It is NOT for downloading 36-bit binaries to 8-bit platforms and uploading them again: TYPE BINARY must be used for that. ckcftp.c, ckuus[5x].c, 20 Dec 2000 (DEC-20 day). Actually there was one tricky part. Suppose the user gives a PUT command without a /BINARY, /TEXT, or /TENEX switch. In this case putfile() does a file scan, and then sets ftp_typ accordingly. But if the prevailing type (FTP TYPE) is TENEX, this would undo it since TENEX is not a file type, but really a transfer mode; scanfile() knows (and should know) nothing about it. So now there has to be a global tenex flag that is set to 0 or 1 whenever the user gives an FTP [SET] TYPE TENEX command. If it's 1, and scanfile() says a file is binary, we set the type to TENEX rather than BINARY. In working through this I also found and fixed a bug with file scanning in FTP, namely that it could alter the global TYPE setting. ckcftp.c, 20 Dec 2000. Changed how global versus local (per-command) FTP TYPE is managed. It was too confusing before. Now we set it whenever an FTP TYPE or SET FTP TYPE command is given, and never reset it. The top-level FTP command entry point restores the global mode every time. Now we can fool with ftp_typ and changetype() all we want and not have to worry about saving and restoring it in 100 places. ckcftp.c, 21 Dec 2000. How to do automatic text/binary switching when downloading in FTP? The server can't do it, but we have the file list in advance so we can use filename patterns. Of course these were originally designed to apply to the local filesystem but at least this way the user has some control (by adjusting the name-pattern list). Added SET FTP GET-FILETYPE-SWITCHING { ON, OFF }. Implementation is simple: for each filename, call matchname() and then changetype() accordingly. ckcftp.c, 21 Dec 2000. Don't set dpyactive if file display is BRIEF. This way we can see FTP protocol messages during file transfer without turning the display completely off. ckcftp.c, 21 Dec 2000. Removed hack installed last May because Latin-1 and CP1252 shared the same index, since now they don't any more. ckuusr.c, 23 Dec 2000. Changed CONST definition to nothing for SCO 3.2V4.x. ckcdeb.h, 23 Dec 2000. Added experimental code allowing Kermit to accept a URL as its first command-line argument, just as it now can accept a filename there. If a URL was detected after cmdlin() has executed, dourl() (in ckcmai.c) is called, which presently handles telnet, ftp, and iksd URLs. FTP and Telnet are handled by switching immediately into the appropriate alter-ego command-line personality; only a few lines of code needed. The IKSD URL does what the "iksget" script does: it logs you in and gets the given file (if a path component was included in the URL; if no path component was included it puts you in CONNECT mode). It writes the iksget script into a malloc'd buffer which it tricks the subsequent code into believing is the "-C" command-line argument. It works fine for anonymous or real-user logins, in the latter case prompting for password if one was not included in the URL, etc. It probably needs a bit more work to account for other authentication methods; I'm not sure what variables to check to tell whether the user is already logged in. It won't log in or get files in NOSPL builds, not that anyone will notice. ckcmai.c, ckuus[4y].c, 23 Dec 2000. Why does \fword({Now , Is, ,the,,time},\%1,{,}) not recognize the two adjacent commas after "the" as an empty word? Because adjacent separators are collapsed (which is what is normally desired, e.g. when spaces are separators). Changing this would be dangerous. But not if we add Yet Another Argument to cksplit() that says whether to collapse adjacent separators, and add the same argument to \fword() and \fsplit(), and leave the default action (when the new argument is omitted) to collapse separators, as now. This is a rather important capability; otherwise we wouldn't be able to parse (e.g.) comma-separated lists that were output by database export procedures. ckclib.[ch], ckuus[r234].c, ckcpty.c, 24 Dec 2000. Added Kermit stuff (cmdmsk, parity, etc) to "telnet -8" semantics. ckuusy.c, 24 Dec 2000. Fix from Jeff to distinguish command-line URLs from DOS filenames. ckuus4.c, 27 Dec 2000. The routine to convert an old-style (pre-CK-6.0) dialing directory to the new format apparently had never been tested in VMS, at least not on an Alpha, where it crashed the program. It worked by accident on the VAX, hard to explain. Fixed in ddcvt(): ckuus6.c, 27 Dec 2000. Fixed HELP SET FILE to also mention SET/SHOW TRANSFER/PROTOCOL. ckuus2.c, 27 Dec 2000. Discovered that mlook() would dump core if called with a constant string, since it lowercased its keyword arg in place. Fixed it to make a copy, in such a way as not to hurt performance -- in fact, it should help a bit. Ditto for mxlook(). ckuus5.c, 27 Dec 2000. Fixed SHO MAC X to say "ambiguous" rather than "not defined" if X is ambiguous. ckuus5.c, 27 Dec 2000. Fixed SHOW COMMAND to include the ON_UNKNOWN_COMMAND definition. ckuus5.c, 27 Dec 2000. The FTP client did not do anything about character sets in filenames. This turns out to be an issue (surprise). People are complaining in the newsgroups that no FTP client lets them get their (e.g.) German-named files from (e.g.) Windows to Unix, using a Unix FTP client and a Windows server. The obvious gotcha is that we can't use UCS-2, period. While this might be OK in file data, it's absolutely not OK in filenames because of the C library and system call APIs. Therefore if the user says the file or server charset is UCS2, we can translate the data but not the filename. (Otherwise, we assume that the file contents and file name use the same character set.) But first... FTP directory listings don't seem get translated, even with: ftp character-set-translation: on ftp server-character-set: latin1-iso file character-set: cp437 Why not? Because the charset args to recvrequest() in doftpdir() were all 0. Fixed in ckcftp.c, 28 Dec 2000. Next, SET FTP CHARACTER-SET-TRANSLATION is OFF even after you SET FTP SERVER-CHARACTER-SET to something, which is kind of disconcerting. So I changed SET FTP SERVER-CHARACTER-SET to also enable translation; why else why else would anybody bother with it? ckcftp.c, 28 Dec 2000. Changed all calls to recvrequest() to provide the appropriate charset arguments. This means that whenever we ask the server to send anything back to us on the data channel, it gets translated. ckcftp.c, 28 Dec 2000. Is this right? For example, NLST tells the server to send us a list of filenames so we can send them back one by one in GET requests. This list is already in its own character set, so why translate it? Answer: because we also need to display each filename on our local screen, and to use it when creating the local file. So when we send each filename back to the server in our GET requests, we must translate it back. And that means our translations better be invertible. But they are, since FTP uses only Unicode as an intermediate character set. Next: Translate filenames in RETR commands. This is done in recvrequest(). The filename is the 'remote' parameter. To do this requires setting up an input helper-function for xgnbyte() and an output function for xpnbyte(): strgetc() and strputc(). strgetc() has to be extra careful about sign extension. ckcftp.c, 28 Dec 2000. First test: get /text /server-char:latin1 /local-char:cp850 Grüße.txt This turns out to be a bit more tricky than expected. First we have to translate the filename in the [M]GET command from local to remote for the NLST command. The NLST results come back in the remote charset, and we translate them to the local one. The MGET loop gets the next filename from this list and calls getfile() with it, which in turn calls recvrequest(). recvrequest() must convert the remote filename back to the remote charset for the RETR command, and then immediately switch its translation around, so the incoming data is translated from the remote to the local charset. This works, but lots of details need ironing out: the name shown in the file-transfer display (plus the fact that curses goes bonkers when asked to display an 8-bit character), plus the name also needs to be translated in the SIZE, MDTM, and other commands. The latter seems to imply that ftpcmd() itself is a better place to put the translation since we also have to do it for CWD, MKD, RMD, etc etc, but ftpcmd() is not designed for this. ckcftp.c, 28 Dec 2000. The change I made to mlook() two days ago inexplicably broke lots of things: IF, FOR, etc. It seems that the upper level code actually DEPENDS on mlook() lowercasing its argument. For now I just put the code back as it was and adjusted SHOW COMMAND not to call mlook() with an uppercase constant string. Figuring out what's really going on will take a quiet day, if I ever have one. ckuus5.c, 29 Dec 2000. Changed ftpcmd() to take 4 args instead of one: the command, the argument string, and the two charset indices. The first argument (the FTP protocol command, such as STOR or LIST) is not translated. The second arg, if not NULL or empty and if both charset indices are greater than -1, is translated from the local charset to the remote one. Changed all ftpcmd() calls (over 80 of them) to the new format, and got rid of almost an equal number of preparatory ckmakmsg() calls, and removed some special-case code I had installed a couple days ago. The new scheme makes the code much simpler, cleaner, and more comprehensible and maintainable. In brief testing, it seemed to work, but needs a LOT more testing, since every single client/server interaction is affected. ckcftp.c, 30 Dec 2000. Having char * s and * p as "global variables for disciplined use" turned out to be too dangerous, got rid of them. ckcftp.c, 30 Dec 2000. Note to myself: bash on the Columbia systems interferes with 8-bit characters no matter what your stty settings are. For any testing involving typing of 8-bit text at the shell prompt or Kermit prompt, use ksh (and then "stty pass8"). I don't know what bash does, but whatever it is, it's so powerful that it prevents Kermit itself from undoing it with it all its ioctls. Also note that "ls" on SunOS doesn't show 8-bit names in files unless you pipe it through another program like cat or more. Now... to set up a testing environment for FTP, I open two windows into the Sun from K95. In both windows, I run ksh and stty pass8. In one window I use Latin-1 as the terminal character set and in the other, DG International. In the Latin-1 window, I have a text file with a Latin-1 name containing Latin-1 characters. I "set file char latin1, set ftp server-char dg" and PUT the file. The file arrives with its name and contents translated to DGI. Then I GET the same file, by name (typing the umlauts, etc, at the Kermit prompt). It returns successfully with the proper name, its contents translated back to Latin-1, except for a few points, indicating the DG->Unicode->Latin1 and Latin1->Unicode->DG translations are not 100% invertible: . Broken bar . Soft hyphen . Macron . Superscript 1 . fractions 1/4 and 3/4 . Thorns and Eths . Division sign . y-acute But this is as it should be, since DGI does not have these characters. The BRIEF-format file-transfer display shows the filename in the local character set, and it also appears that way in the transaction log. (In the curses display, however, 8-bit characters appear with the 8th bit truncated, but highlighted, so "Gre" appears as bold "Gr|_e"; there's not much Kermit can do about that. It will no doubt be different with more modern termcap/terminfo/curses/ncurses implementations.) The first test "pushed" a Latin-1 file to the DG side and "pulled" it back again. A second test pulled a DGI file from the DG side and pushed it back. This worked fine too, but this time of course the characters that were lost in the text were those which DGI has but Latin-1 lacks: various daggers, less/greater-equal signs, Florin, arrows, OE digraph, etc. The third test added binary files that had 8-bit names. For this, some small fixes were necessary, most of them due to the fact that we turned translation off when TYPE was not ASCII. After this, we can "mput *" where * matches a mixture of text and binary files, and each file is sent in the appropriate mode automatically, and each file's name is translated, regardless of mode. ckcftp.c, 31 Dec 2000. However, a command such as "mget Gre.*" doesn't work. Kermit translates the name correctly (as shown in the debug log), but the server (wu-ftpd) says no files match. However, "mget G*.*" works fine -- the files come, their names are translated, and name-based text/binary mode switching works fine. This must be a bug in wu-ftpd, since "get Gre.txt" works; evidently the combination of 8-bit characters and wildcards is fatal. Tried the same thing to the VMS UCX 2.0 FTP server: "File specification syntax error". Enclosing in quotes only changes the message to "Error in filename". Evidently this one simply can't deal with 8-bit names. But the same test works fine with NcFTPd, which I found on an IRIX system. So Kermit is OK. Changed cmdmsk (SET COMMAND BYTESIZE) to be 255 rather than 127 by default. After all, tomorrow is 2001... ckucmd.c, 31 Dec 2000. Problem: getreply() does not translate FTP server reply messages. This means that "Gr<8-bit-chars>*.txt not found" won't be translated back into the local charset for display, and also that FTP VDIR doesn't translate. But getreply() is pretty horrendous... But oops, this also affects PWD, plus replies to MKDIR, RMDIR, etc. Guess I'd better fix it... (tomorrow). Meanwhile, I tested charset translation with directory names: MKDIR, CD, PWD, DIRECTORY, DELETE, and RMDIR. All OK except for non-translation of replies. Updated copyrights to 2001: ck[cuw]*.[cwh], makefile, COPYING.TXT, 1 Jan 2001. Back to getreply()... First we add charset index args to the function itself, its prototype, and all invocations. Next we find the spots in getreply() where the server response is output. As far as I can tell, these are the two instances of printf("%c",c). When translation is active, a new routine, xlatec(), is called instead of printf(). The new routine must be mindful of the possibility the input and output sizes can be different; thus an input byte might be just part of a character (e.g. one octet of a UTF-8 sequence), and ditto for the output, in any combination. So xlatec() has to maintain an internal catch-up buffer and act somewhat asynchronously. ckcftp.c, 1 Jan 2001. Fixed several FTP glitches: . RPWD translation didn't work. . FTP VDIR did't translate . RDEL overprinted with "...OK". The new code seems quite solid. I ran through all the previous tests with no problems, and then I did them again with UTF8 on the far end. ckcftp.c, 1 Jan 2001. cmdate() parsing of TODAY, YESTERDAY, and TOMORROW has been broken since the strncpy() -> ckstrncpy() rampage of last June. Fixed by reverting the ckstrncpy()'s in these sections back to strncpy()'s. Remember: in ckstrncpy(), the number refers to the size of the buffer -- the whole source string is copied up to the given size. In strncpy(), the number refers to the size of the source string, not the buffer. ckucmd.c, 2 Jan 2001. Added SET FTP DATES { ON, OFF }, and accounted for it in SHOW FTP and HELP SET FTP. ckcftp.c, 2 Jan 2001. Changed chkmodtime() to have a third argument, fc (function code), and changed all four invocations of it to include it (0). 0 means compare file times, as before; 1 will mean set local file's modtime from the remote. I still have to fill in code to actually perform the new function. ckcftp.c, 2 Jan 2001. Fix from Jeff for TAKE path, plus additional SET MODEM debugging. ckuus[r34].c, 3 Jan 2001. Added appropriate #ifdefs around dologend() calls in ckucns.c and ckucon.c. 3 Jan 2001. Finished FTP file-date setting. FTP DATES is OFF by default, for compatibility with existing FTP clients and so as not to mislead, since even if we can set dates of incoming files, there is no way to tell the server to set the date of a file we send to it, and also because the server might not support MDTM. Two new functions were added. First mkutime(), which does what timegm() does, either by simply calling timegm() itself if HAVE_TIMEGM is defined, or by executing the equivalent code supplied by Russ Allbery of Stanford (author of INN) if HAVE_TIMEGM is not defined, which it is not by default. As far as I can tell by testing with both a recent file and a 15-year-old file, locally and across timezones, Russ's code is equivalent to timegm(), at least on SunOS. The second new function, setmodtime(), simply hides all the hideous #ifdefs and differences from the rest of the code; it was copied direct out of zstime() in ckufio.c; it could be consolidated if necessary, but for the present I'd rather keep the FTP module as self-contained as possible. ckcftp.c, 3 Jan 2001. Fixed recent FTP code to be K&R-safe. ckcftp.c, 4 Jan 2000. Added support for SCO OSR5.0.6. makefile, ckuver.h, ckcnet.c, ckutio.c, 4 Jan 2001. After nearly completing the build-all I noticed a couple bad problems with FTP. First, commands like RPWD showed their results only if the debug log was open. This was due to excessive cleverness in getreply() for the benefit of VDIRECTORY. Fixed by removing the special case. ckcftp.c, 5 Jan 2001. Second, GET path/file didn't work. It never did. Here's why: if you send NLST to wu-ftpd, it returns a recursive list. Therefore, Kermit, when going through the NLST file, silently skipped past any file that included a path segment if /RECURSIVE was not given. But this ignored the possibility that the user included a path segment in the [M]GET filespec. Taking this into account required a rather major rewrite of the [M]GET code. The trick is: if the user included a path segment in the [M]GET command, we have to check each remote filename to see if it starts with the same path segment, and if so, strip it before creating the local filename; this must be done whether /RECURSIVE was given or not. In addition, if /RECURSIVE was NOT given, we skip any file that still has a path segment after stripping the one the user gave, because in that case the server is sending a recursive list when we did not ask for one. NOTE: whether the server returns a recursive list, and exactly what it includes, is highly dependent on the format of the user's filespec. Suppose "foo" is the name of a subdirectory of the (UNIX) FTP server's current directory. Then "mget foo", "mget foo/", "mget foo/*", "rcd foo, mget *", and "rcd foo, mget ." might each have entirely different results. ckcftp.c, 5 Jan 2001. Changed SET FTP DEBUG ON to report the name of the NLST temp file and to not delete it. ckcftp.c, 5 Jan 2001. FTP CHMOD was broken in ftpcmd() rewrite; the filename arg was lost (two places). ckcftp.c, 5 Jan 2001. After giving an RDELETE command, we stopped printing responses. Mishandling of the 'silent' variable. ckcftp.c, 5 Jan 2001. FTP setmodtime() didn't initialize its stat() struct prior to reference. Also, it didn't include the necessary utime() related header files. ckcftp.c, 5 Jan 2001. So I repeated the mini-build-all (takes about 3 hours), and then "ftp put filename as-name" suddenly started dumping core (and I had been using this command all day). And not because any code changed either. Turns out the problem was in a debug statement, one of whose parameters made a bogus array reference: even though debugging was not on, the array reference was still resolved, and that's what caused the crash. It's been like that since Day 1. Fixed in doftpput(), ckcftp.c, 5 Jan 2001. Another new problem resulting from today's fixes: "get foo bar" gave an error, "?As-name for MGET must contain variables such as \v(filename)". But it's GET, not MGET. Fixed in doftpget(), ckcftp.c, 5 Jan 2001. AOS/VS doesn't have [nh]to[hn]l(). Built OK on AOS/VS but it seems to be too big to run on the little MV 2500 -- mallocs fail, etc. ckctel.c, 6 Jan 2001. If you start Kermit on AOS/VS without its init file, it runs OK and can make a Telnet connection, but crashes in scanfile(). Hmmm, there's no #ifdef to get rid of scanfile(). So I used #ifdefs to set file scanning off for DG, and to remove the SET FILE command that turns it back on. ckcmai.c, ckuus7.c, 6 Jan 2001. #ifdefs for SCO XENIX 2.3.4. The Xenix ftp client works, but only if you "set ftp passive off"; otherwise it hangs in the connect() call. ckcftp.c, 6 Jan 2001. ---7.1.199 Alpha.02--- Changes from Jeff 4-20 Jan 2001:: . ckcmai.c: fix to isabsolute() for recognizing PC disk letters. . ckuusx.c: add tthang() to doclean(). . ckudia.c: increase size of modem-message buffer and make length a symbol. . ckcfn3.c: temporarily restore verbose debugging to getrbuf(). . ckuus4.c: fix spelling of \Fminimum(). . ckcfns.c: fix a repeat-count boundary condition (see CHECK THIS). . ckcnet.c: move some http code. . ckuus3.c: fix dial directory lookup and dial-command-string nullification. . ckcfn2.c: temporary debugging for "receive window full" errors. . ckcftp.c: "testing=0"; URL use GET not MGET; make FTP DEBUG take precedence over FTP QUIET; Changes from me that Jeff never picked up: . ckucns.c: conditionalize dologend() calls. . makefile: all the updates for Alpha.02. Jeff switched the cmdlinget() code to do GET rather than MGET because some server was not returning proper information when an FTP URL was used. But this code is used not only for FTP URLs, but also for regular command-line driven GETs -- it's a new, documented, and very handy feature of the Kermit FTP client, and it works fine. So if there is some problem when doing this with URLs, then cmdlinget() should make the distinction at runtime -- if it's a URL, do a GET, otherwise do an MGET. Changed Jeff's change accordingly. cmdlinget(): ckcftp.c, 21 Jan 2001. I still don't like it though, since FTP URLs with wildcards work fine with most servers, and this is a big advantage for Kermit over other FTP URL interpreters, especially since Kermit does the automatic text/binary switching. It seems a shame to disable this feature for everybody just because one server somewhere misbehaves when we use it. Actually, now that I think about it, the server has no idea whether the client used an FTP URL or gave an MGET command interactively -- the result would be the same. I checked this by making an interactive connection to ftp.ssc.com and doing "mget pub/lj/listings/issu72/*" and sure enough, it did not work (in response to NLST, the server returned basenames, not full or relative pathnames, so each RETR command based on the NLST list resulted in "file not found"). How can we possibly work around this problem? FTP URLs don't leave any room for options. OK, here's what we do: if the server OS is UNIX, we can check the filespec for wildness. If wild, we do MGET; if not we do GET. This seems to do the trick. Of course it still doesn't let us MGET files with a pathname from the broken server. cmdlinget(): ckcftp.c, 21 Jan 2001. A Receive Window Full error occurred when the client's GET packet was answered by a retransmission of the server's ACK to the client's I packet (the exact circumstances were pinned down by Jeff). Fixed in ckcpro.w, Y, 22 Jan 2001. From Jeff: Enable HTTP and shadow passwords for SCO ORS5.0.6; remove temporary debugging for receive window full; ckcdeb.h, ckcfn[23].c, ckcnet.c, 4 Feb 2001. The USPS reported that the session log came out differently in interactive (CONNECT) and scripted (INPUT/OUTPUT) sessions in VMS. Fixed in ckuusx.c, 4 Feb 2001. Fixed HELP SET BUFFER text to say "Kermit" instead of "C-Kermit". Fixed typo (s2 instead of s3) in HELP FUNC SUBSTITUTE. Removed spurious second arg from HELP FUNC FPINT text. ckuus2.c, 4 Feb 2001. Made \fmodulus() visible again, since I can't recall why I made it invisible. ckuus4.c, 4 Feb 2001. If the user says SET PREFIXING or SET CONTROL PREFIX , then also implicitly set CLEARCHANNEL (clearrq) to OFF; otherwise the user's command has no effect at all. Evidently we had done this before, but then backed off. The trick is to not do it in setprefix(), but rather in the command-parsing code, so that way it only occurs if a SET [CONTROL] PREFIX command is given explicitly. ckuus3.c, 4 Feb 2001. Discovered there was no way to make UNIX C-Kermit convert spaces in incoming filenames to underscore (or anything else). Fixed in nzrtol(), ckufio.c, 4 Feb 2001. At Jeff's suggestion, reordered the statements at E, which looked like they might free the wrong sequence slot. ckcpro.w, 4 Feb 2001. Add missing #ifdef DEBUG..#endif around an unguarded reference to debtim in ckcftp.c, to fix Ultrix build which happens to include -DNODEBUG. 4 Feb 2001. Mark Sapiro noticed that the fullscreen file transfer display, in its summary message at the end of a transfer, reported the total number of files, rather than the actual number that were sent (the new code forgot to subtract the number that were rejected). Fixed in screenc(): ckuusx.c, 4 Feb 2001. Changed command-line ftp handler to exit and fail if it can't make the connection or log in, rather than going ahead as if it had. ckuusy.c, 4 Feb 2001. Fixed a mistake introduced in my 21 Jan 2001 edit of cmdlinget(), which caused it fall thru to MGET after doing a successful GET. ckcfns.c, 4 Feb 2001. Got rid of warning messages about STRU, MODE, and REST. ckcftp.c, 4 Feb 2001. PeterE discovered that MKDIR didn't give an error message when it should have under certain extremely bizarre circumstances (namely, C-Kermit had been started with an init file that defined a macro that returned a string and then SET PROMPT to \fexec() of that macro). Fixed in ckuusr.c, 5 Feb 2001. The following: ftp ftp://ftp.ssc.com/pub/lj/listings/issue73/\*.tgz accesses an FTP server whose NLST command does not return the path, thus MGETs containing pathnames fail. On Jan 21, I added a workaround for (single-file) GETs. Today I added similar (worse) code to handle MGETs: if (a) the server is UNIX, and (b) an NLST name does not contain a '/', and (c) the MGET name *does* contain a '/', then I append the NLST name to the MGET name after the rightmost slash before sending the RETR request. I do this for both MGET commands and command-line GETs and URLs. It works with ftp.ssc.com server and also still works with the more common servers that do include paths in NLIST names. ckcftp.c, 5 Feb 2001. Some minor adjustments of text-patterns. ckuusx.c, 5 Feb 2001. For docs: describe effect of -d for FTP command-line GETs/PUTs. Temporary extra debugging for doinput(): ckuus4.c, 6 Feb 2001. Vitek Pepas complained that the "Closing /dev/blah..OK" message came out when Kermit exits, even if QUIET is ON, if the EXIT command is executed from a script. Why? Because the QUIET setting is on the command stack. EXIT pops the command stack. If SET QUIET ON was given in the command file, it's lost when the stack is popped. I worked around this by having doclean() check msgflg rather than quiet. (The right way to fix this would be to make sure popclvl() wasn't called before checking QUIET, but the workaround is safer.) ckuusx.c, 6 Feb 2001. Wasted some time trying to make an Irix 6.5 compiler warning about utime() go away. At least I reduced the warning to its most absurd form: Argument of type "const struct utimbuf *" is incompatible with parameter of type "const struct utimbuf *". The odd thing is the exact same code is used in ckufio.c, but in that case there's no warning. ckcftp.c, 6 Feb 2001. Added #ifdefs to choose whether to use mktemp() or mkstemp() to create temporary files. For now we use mkstemp() in Linux and 4.4BSD, since those are the platforms that warn us at link time that mktemp() is unsafe. Now it builds with no warnings on RH7.0. ckcdeb.h, ckcftp.c, 8 Feb 2001. From Jeff: replace some duplicated chunks of K95 terminal-dimension setting code with calls to os2_setterm{width,height}(): ckuus[37].c; fix INPUT for K95 to ignore spurious keyboard events: ckuus4.c, 20 Feb 2001. DELETE switches accidentally omitted /[NO]PAGE. Fixed in ckuus6.c, 20 Feb 2001. Made SET FILE T be an acceptable abbrev for SET FILE TYPE, as it used to be before TEXT-PATTERNS was added, so the book doesn't lie. ckuus7.c, 20 Feb 2001. Made VERSION show version, LICENSE (= COPYRIGHT) show copyright notice and license. ckuus[r2].c, 20 Feb 2001. Fixed many HELP messages not to say C-Kermit if it's K95. Also changed HELP SET COMMAND not to say the default COMMAND BYTESIZE is 7. Also fixed HELP SET PROMPT text for K95. ckuus2.c, 20 Feb 2001. Changed default TERMINAL BYTESIZE from 7 to 8 (cmask from 0177 to 0377). This is appropriate for most situations, but hmmm, SunOS 4.1 sends even parity when you log in on a serial port. But hardly anybody logs in on a serial port any more so no big deal. ckcmai.c, 6 Mar 2001. Changed default modem type from NONE to GENERIC. ckcmai.c, 6 Mar 2001. Changed GENERIC modem definition to not send any init string, and changed the name from "generic-high-speed" to simply "generic" (but still accept "generic-high-speed"). ckudia.c, 6 Mar 2001. But if the default modem type is not NONE, it no longer makes sense to have the default MODEM HANGUP-METHOD be MODEM-COMMAND, because if you make a direct connection, hanging up sends ATH0 or whatever. So I changed it to RS232-SIGNAL. ckudia.c, 6 Mar 2001. Updates from Jeff, mainly for GUI: ck_ssl.c, ckcnet.c, ckuus7.c, ckuus5.c, ckuusx.c, ckcmai.c, ckuus3.c, ckcdeb.h, 7 Mar 2001. The DIRECTORY /RECURSIVE command and the \frdir() function always follow symlinks. This is bad for many reasons, such as multiple traversals of the same tree, and even endless loops. Sometimes you want to follow links, sometimes you don't. Ditto for SEND. So there needs to be a switch. Added DIR_LNK/DIR_NLK and SND_LNK/SND_NLK to ckuusr.h. Added /NOFOLLOWLINKS (and /FOLLOWLINKS) to DIRECTORY command; this uses the same underlying mechanism that I added to make DELETE /RECURSIVE never follow links. Changed DIR to not follow links by default, since following links is dangerous and not following them isn't. domydir(): ckuus6.c, 7 Mar 2001. Same deal for SEND. ckuusr.c, 7 Mar 2001. Fixed \frdirectory() to never follow links. ckuus4.c, 7 Mar 2001. New SunOS 4.1 secure builds from Jeff. ck_ssl.c, makefile, 9 Mar 2001. Updated HELP SET { COMMAND, TERMINAL } BYTESIZE text. Discovered that \fday() dumps core when given a date prior to 17 Mar 1858. Fixed in ckuus4.c, 13 Mar 2001. Fixed cmifi2() (and its clients, e.g. cmifi(), cmdir(), etc) to return -3 if given {} or "". ckucmd.c, 13 Mar 2001. Updated help text for SET TERM BYTE, SEND, DIR, ckuus2.c, 13 Mar 2001. There is something in ckcftp.c that I never understood, but didn't cause any trouble until now: #define sig_t my_sig_t #define sigtype SIGTYP typedef sigtype (*sig_t)(); In Ultrix 3.0 all hell breaks loose. In all our other signal-using modules we use only SIGTYP, so I changed ckcftp.c to follow the same conventions, and removed all references to sigtype and sig_t. At least this makes ckcftp.c consistent with the other modules. But it made no difference at all. For some reason the Ultrix compiler violently objects to "oldintr != cmdcancel" in getreply() because the operands are not the same type (but they are). So I backed off the changes and then #ifdef'd this comparison away for Ultrix 3.x only. makefile, ckcftp.c, ckuver.h, 15 Mar 2001. The change I made to \frdir() a week ago had a typo that broke recursion. Fixed in ckuus4.c, 17 Mar 2001. A couple minor cosmetic changes to DIAL module, ckudia.c, 17 Mar 2001. Updated sourced from Jeff: shotcs(): ckuus5.c; add brace stripping to SET FILE DOWNLOAD-DIR: ckuus7.c, 17 Mar 2001. Added new modem types: . lucent: Lucent Venus chipset . pctel: PCTel V.90 chipset . conexant: Conexant (ex-Rockwell) modem family . zoom-v32bis: New name for "Zoom" . zoom-v34 Zoom V.34 . zoom-v90 Zoom V.90 56K . zoom-v92: Zoom V.92 with V.44 data compression . zoltrix-v34: New name for "zoltrix" . zoltrix-hsp-v90: Synonym for PCTel . zoltrix-hcf-v90: Synonym for ITU-T-V250 . smartlink-v90: Synonym for usrobotics (same chipset) . acer-v90: Synonym for Rockwell-v90 All untested. ckuusr.h, ckudia.c, 18 Mar 2001. Merge with Jeff's updated sources: ckcnet.c ckudia.c ckuath.c ckuusr.h makefile ckuscr.c ckctel.c ckufio.c ckcftp.c ckcnet.h ckuus4.c ckuus7.c, 20 Mar 2001. Tighten up generic dialing a bit. Don't bother with speaker settings. Change dialhup() to not send +++ATH0+++ if modem type is generic AND it's a serial connection. ckudia.c, 20 Mar 2001. Changed defaults for MODEM ERROR-CORRECTION, DATA-COMPRESSION, and HANGUP-METHOD to match new default modem type of GENERIC, so that start-up defaults are the same as when you get when you SET MODEM TYPE GENERIC. ckuus3.c, ckudia.c, 21 Mar 2001. Fix from Jeff to parsedir() for parsing directory-path lists. ckuus3.c, 22 Mar 2001. Added IF GUI, to allow portable scripting of GUI-related commands, such as font selections (e.g. "if ( k-95 && gui ) set font courier_new 16"). Maybe now we should get rid of "k95g" as a \v(program) value; I don't like it because it's likely to break existing scripts. Now we can use IF GUI to test for GUI-ness without disrupting anything. Also removed a few more stray references to Kermit/2. ckuus[26].c, 22 Mar 2001. Removed &Sn from V.250 init strings - it's not in the standard. ckudia.c, 25 Mar 2001. Sync'd files with Jeff; his changes mainly for GUI font selection. 27 Mar 2001. In response to scattered but persistent requests over the years, added SET SESSION-LOG TIMESTAMPED-TEXT. It's done sys-independently in logchar() so should work for all platforms. The command is cleverly structured to allow timestamps only for text-mode logs. The timestamp at the beginning of each line shows the hh:mm:ss.nnn when the first character of the line appeared. ckuus[23x].c, 27 Mar 2001. Also changed ckucns.c and ck[uv]con.c to say "Session Log: filename, timestamped-text" (or other type), but it's just a frill, not necessary in all CONNECT modules. 27 Mar 2001. Changed askmore() to accept G (Go) or P (Proceed) to show all the rest without asking. ckuusx.c, 27 Mar 2001. Started working on an FTP NLST command but didn't finish. The idea is to be able to make the NLST temporary file available programmatically, as discussed on the newsgroup a while back. This will require (a) not deleting it (done); (b) not actually GET'ing the files (done); (c) putting its name in a variable; and (d) not creating a new file for each filespec in the command (e.g. "ftp nlst a* b* c*" should create one temp file, not three). Invisible for now; will finish later. ckcftp.c, 27 Mar 2001. Small correction to timestamped logs (session and debug) for the case where ztime() does not set ztmsec > -1. ckuusx.c, 28 Mar 2001. Backed off on yesterday's FTP changes. I should take more Vitamin E... The NLST command was already there, as "mget /namelist:filename ...". I really don't remember putting this in. ckcftp.c, 28 Mar 2001. Made SET SESSION-LOG TIMESTAMPED-TEXT un-invisible for all, since it didn't turn out to be UNIX-specific like I thought it would be. ckuus3.c, 29 Mar 2001. Added FTP OPEN /PASSIVE and /ACTIVE. ckcftp.c, 29 Mar 2001. Changed getyesno() interpretation of 2nd argument. If it contains 1, "quit" is a valid keyword. If it contains 2, "go" is a valid keyword. ckuus3.c, 29 Mar 2001. Changed DELETE /ASK to call getyesno(xxx,3) to allow both "quit" (cancel the DELETE command) and "go" (delete all the rest of the files without asking). ckuus[26].c, 29 Mar 2001. If C-Kermit in server mode is given a REMOTE CD command, it would print the directory name. Fixed in docd(): ckuus5.c, 29 Mar 2001. Refixed a typo in ck_copyhostent() in ckcnet.c that broke non-ANSI builds. Put ck_copyhostent() and all references to it in #ifdef HADDRLIST..#endif in ckcnet.c, ckuath.c, ckufio.c, and ckcftp.c. Fixed some char/CHAR complaints in ckuusx.c. Added makefile entry and designer banner for FreeBSD 4.3: makefile, ckuver.h. Added missing "extern int sessft" in ckvcon.c. On my own SCO 5.0.5 system, compilation fails unless I add -DDCLTIMEVAL, and then getspnam() is missing at link time. So I changed ckcdeb.h to not automatically define CK_SHADOW for CK_SCOV5 if NOSHADOW is defined, and added -DDCLTIMEVAL -DNOSHADOW to KFLAGS, and it builds OK, but with lots of warnings about getsockopt() and other function args. But if I build on Boyd G's 5.0.5 system, there are no errors and no warnings. ---7.1.199 Alpha.03--- The ck_copyhostent() routine dumped core on many platforms, Jeff fixed it. Unfortunately, this happened just after the 3-day build-all for Alpha.03, but before the announcement. I updated the sources with Jeff's changes: ckcdeb.h, ckcmai.c, ckuusr[c57].c, ckcnet.c, 1 Apr 2001. Upped the test version from Alpha.03 to Alpha.04. ckcmai.c, 1 Apr 2001. While I was at it... the FTP client has been spewing out more and more messages when logging in to the server. To avoid this I had set ftp_vbm = 0 around the initial connection. Now it seems that ftpcmd() always sets ftp_vbm back to ftp_vbx before exiting, so setting ftp_vbm to 0 and then calling ftpcmd() several times does not suppress the messages as expected. Too many cooks syndrome. Now the code is full of little tricks to force verbose mode on or off around certain commands, to undo the forcing, and undo the undoing of the forcing, to the extent that it's incomprehensible and the SET FTP VERBOSE command has no predictable effect. So I got rid of the tricks and set FTP VERBOSE-MODE OFF by default. Now the command does what it says it does and the user doesn't get tons of cryptic messages. ckcftp.c, 1 Apr 2001. Built new binaries for SunOS, Solaris 2.5.1, Solaris 7, Unixware 7, RH 7.0, Freebsd2,3,4, and a bunch of others, but not the slow ones. There was no way to install the new binaries on watsun alongside the old ones because /pub/ftp too full, so installed them on ftp.kermit, then pruned old binaries, and then removed all ck* from watsun's kermit/test/bin/ and copied ftp.kermit's there. This dual-server business is getting rather convoluted; now watsun is the master for some directories but ftp.kermit is the master for others. We'd better totally cut over soon... ---7.1.199 Alpha.04--- Changed FTP ftpcmd() and getreply() to have a new verbosity parameter, to simplify switching forcing server-reply displays on or off in certain contexts. Changed VERBOSE default back to ON and used the new parameter to hide the initial connection dialog (REST, MODE, STRU, SYST). ckcftp.c, 2 Apr 2001. Added aix43x25 makefile entry. makefile, 6 Apr 2001. From Jeff: . Add SET PORT /SHARE for K95. . Fix SRP FTP bug. . Fix bug with generic dialing pausing for several secs. . Fix IKSD prompt to send rather than just . . Fix dolognet() not to write into uidbuf[], which was preventing SET HOST /USER:{}, SET LOGIN USER {}, etc, from working. . Create separately callable getexedir(), extracted from prescan(). . Restructure FTP module for cleaner message control. ckcdeb.h, ckcmai.c, ckcnet.c, ckuus[347x].c, ckudia.c, ckcftp.c, 7 Apr 2001. FTP messages need some tuning. Data transfer commands like DIR, GET, etc, are too verbose. But this would get us into the editing/censorship business again. I think what I need to do is add SET FTP VERBOSE-MODE NORMAL. This will be the "edited" mode that shows some messages but not others, like a regular FTP client. After that ON will show all messages, OFF will show no messages, and NORMAL (the new default) will show selected messages. Jeff and I added CONNECT /ASYNC, plus some other minor updates from Jeff: ckcker.h, ckuusr.h, ckcmai.c, ckuus[r47].c, ckudia.c, ckcftp.c, 9 Apr 2001. Lots of changes from Jeff from last several weeks, 27 Apr 2001: Misc small adjustments: ckcmai.c ckcdeb.h ckcker.h CONNECT /ASYNC, APC details: ckuus[r457].c ckcfn2.c Security: ck_ssl.[ch], ckuath.c SET PRINTER fixes for K95: ckuus3.c New generalized HTTP CONNECT: ckcnet.h ckuusr.c ckcftp.c Fix modem capabilities AT testing: ckudia.c Added \v(dm_hf) (dial modifier hook flash) and \v(dm_wb) (dial modifier wait for bong). ckuusr.h, ckuus4.c, ckudia.c, 27 Apr 2001. Added --version command-line switch. ckuusr.h, ckuusy.c, 27 Apr 2001. Trimmed trailing blanks and folded long lines, most modules. 27 Apr 2001. This will be Beta.01 after build-all and more cleanups. Fixed an oversight in merging Jeff's SET PRINTER code from yesterday. ckuus3.c, 28 Apr 2001. While helping a user I discovered that C-Kermit on NetBSD always fails to hangup by dropping DTR; the tcsetattr() call always fails with EINVAL. It doesn't like an argument of B0. The same code works fine in FreeBSD. Found out that the NetBSD tty device driver code is broken (for some reason the B0 hangup code is commented out). The workaround is to use ioctl() with TIOCCDTR and TIOCSDTR -- Yet Another Hanging Up API. Added code for this to tthang(), tested in NetBSD (which uses it) and FreeBSD (which doesn't). On BSD44 and POSIX systems, the use of this code can be forced by defining USE_TIOCSDTR. ckutio.c, 28 Apr 2001. Another correction to ckuus3.c (I left out a line AGAIN in yesterday's correction). OK, no more allergy pills for me. 29 Apr 2001. Cleanups resulting from HP "flexelint" run on Alpha.04, as well as gcc -Wall. A couple genuine (potential) problems were found, marked with "+": General (all or many modules): "Constant value Boolean": These warnings are bogus - the constructions are deliberate. "Variable not subsequently referenced" and "not accessed": These are harmless, and result from the tangle of #ifdefs. "Fixing" these warnings would introduce even more spaghetti and risk serious breakage, except in very short functions. "Control flows into case": these are all deliberate. "Expected a constant" - Bogus; they are constants, e.g.: extern CHAR (*xls[MAXTCSETS+1][MAXFCSETS+1])(CHAR). "Array subscript has type 'char'": yes, it does, by design. Thousands of warnings about omitted braces around initializers, harmless. Warnings about signal(): "Error:64:Type mismatch (arg. no. 2) (void (*)(int) = int (*)())": Many modules. Lint is simply not resolving all the preprocessor definitions and prototypes correctly. All these statements compile without complaint in the optimizing compiler and execute correctly. Added extra parens around "while (*p++ = *s++)..." etc to avoid warnings from gcc -Wall. Ditto for boolean expressions using && and || where gcc "suggests" parens for grouping, as if the rules of precedence can't be trusted. Well, who knows, maybe they can't. Cast non-char* args to free() to (char *) to shut up stupid warnings. ckcfn2.c: + 1328: mydata[i++] = tochar(chk1(mydata+lp,i+lp)); (Depends on order of evaluation, fixed by using a temp variable). Cast non-(char *) args to free() to (char *). Various other minor syntax adjustments from gcc -Wall. ckcfn3.c: Various type mismatches -- casts added, etc. ckcfns.c: There were some bogus out-of-bounds array reference warnings in nxtdir(), which could never have happened in real life but I fixed them anyway. ckclib.c: + chartostr(): a true out-of-bounds array reference was caught: if this function was ever called with a char val of 128-159, it made a wild reference. makestr(): Out-of-bounds pointer in makestr() has to be bogus. ckregex(): adjusted some "suspicious casts". ckregex(): Out-of-bounds warning is bogus. ckcmai.c: added a few missing prototypes. ckucmd.c: added some casts and extra parens. ckucns.c: Lots of bogus "expected a constant" warnings; they *are* constants. Added missing prototypes. + if (printing & escbuf > 1) should have been "&&" (probably same effect). Moved dologshow() prototype from ckuusr.h to ckcker.h where ckucns.c can find it. ckucon.c: All warnings inconsequential, and HP-UX doesn't use this module anyway. ckudia.c: Out-of-bounds references to tonecc[] array: can't happen (read the code). Ditto for pulsecc[]. Out-of-bounds reference to pp[] array: fixed off-by-1 declaration. ckufio.c: ctime() "Should be a pointer". It is a function that returns char *. zinfill() "loss of information": bogus, anyway it's only for debugging. "Non-existent return value for write()" -- that's just silly. "Redefinition of symbol 'zchki(char *)'" -- bogus, lint is wrong; it's not picking up the definitions from ckcdeb.h. Ditto for nzrtol() and nzltor(). zchkpid(unsigned long pid): "Expecting an identifier or other declarator": apparently "pid" must be a reserved word? Changed it to xpid. A similar warning at line 2848. There's nothing wrong with these lines; lint is confused. zhomdir not initialized: wrong, it is. fp[filnum] = popen(...) "type mismatch": wrong; it's not. 3073/1276: 2 vs 3 args to open(). This is legal, see "man 2 open". fp[filnum] = fdopen(...) "type mismatch": wrong; it's not. "Redefinition of symbol zxpand()": Wrong. 3437: "flags not initialized": nonsense, it's a parameter to this function. 3621+ff: out-of-bounds ref to p[]: No, it can't be out of bounds. 3647: type mismatch: no it isn't. Lint is not picking up the prototypes. 5724: read arg #3: lint doesn't understand sizeof. 6349: wrong number of args to execl(): wrong. See "man execl". Complaints about ckstrncpy() unfounded. ckuscr.c: Possible out-of-range array ref: bogus, but I increased the array size. ckutio.c: 10991: One bad call to congetbuf() was found (missing arg) but it wasn't in the HP-UX code. 3995: read() is called with a long; cast it to int. We'll always get warnings about read() arg #3 (int vs unsigned int vs size_t etc), but fixing them is impossible, and they only really matter when size_t is not int or unsigned int, or if read really wants a long or unsigned long. All rest are because Lint didn't pick up function prototypes, preprocessor symbols, and/or typedefs. It's even reading big blocks of code that are not selected for HP-UX. ckuusr.c: 700 "omitted braces" warnings safely ignored. Some free((char *)blah) casts added. ckuus3.c: Missing braces around keyword table entries: ignored, harmless. + 2102: pp not init'd in parsedir(). Sure enough, it wasn't! 3574ff: Possible division by 0. No, these are all prechecked. ckuus4.c: Missing braces around keyword table entries: ignored, harmless. Complaints involving sizeof: bogus. Complaints about signal(): ignored. Many xx.os_specific = '\0'; changed to xx.os_specific = ""; 6614: possible out-of-bounds array ref: no, prechecked. 6749ff: ditto. 6810ff: ditto. 6898ff: ditto. 7036ff: ditto. 7321ff: ditto. 9214, incompatible types: wrong, Lint missed the prototype. 12904ff: possible out-of-bounds array ref: no, prechecked. ckuus5.c: + 1890: while () condition lacked parens for proper grouping. 3602: out-of-bounds: bogus. 3674: ditto. + 3919: This one was interesting... Evidently \%x variables are not limited to just letters and digits. Any ASCII character can be used in the 'x' position, e.g. \%_, \%=, \%+, etc. I did this so long ago I forgot about it. Anyway, there could have been an out-of-bounds reference if anybody took advantage of this undocumented feature and defined a \%~ variable. Fixed in ckuusr.h and ckuus5.c. 3939: Warnings about a_ptr[] references, bogus (prechecked) but I increased the dimensions anyway to suppress the warnings. 4151ff: like 3919. + 8003ff: possible out-of-bounds references to parameter s[]: fixed. ckuus6.c: 4797ff: zfcdat() is guaranteed to return a string in the right format. But I added code to defend against improper zfcdat() implementations anyway. 6475ff: out-of-bounds: bogus (prechecked). + 8895: An out-of-bounds reference was possible here (to s[3]), fixed. 8903: Out-of-bounds reference warning: bogus; it's OK to reference the terminating NUL. ckuus7.c: 2346ff bkupnum(): array ref out-of-bounds, bogus (prechecked) 11827: xx.os_specific = '\0'; changed to xx.os_specific = ""; ckuusx.c: 4485: *p ref possibly out of range: no. ckuxla.c: 274 "omitted braces" warnings (safely) ignored. 53 "expected a constant" warnings (safely) ignored. "Too many initializers for aggregate", "expecting a '}'": nonsense. Also: Added missing prototypes for do_pty() and end_pty() to ckcdeb.h. Removed unused variables from functions where it was safe to do. (Apr 29 - May 1, 2001.) Added prototypes for all the IKSD database routines to ckcker.h, 2 May 2001. Removed an extraneous trailing arg to slotstate() call. ckcpro.w, 2 May 2001. Added prototypes for ftp routines called from outside to ckcdeb.h, 2 May 2001. Recent modem-type additions included some aliases that made modem-type display confusing; for example, "set modem type usrobotics" would list the modem type as "smartlink-v90" because that was the first one in the list (alphabetically) that had the USR type. Added an M_ALIAS flag for the keyword list to prevent this. ckudia.c, 2 May 2001. Fixed mlook() and mxlook(), which actually wrote into their argument strings. This was discovered last December but tabled: Something is very wrong here. The OLDMLOOK code, which was used prior to 28 Dec 2000, lowercases its argument string IN PLACE. But fixing this to make a copy of the argument string and lowercase the copy breaks just about everything. So evidently the higher-level code depends on mlook() lowercasing its argument. For now we just put it back the way it was before. Fixing this will require a detailed audit of mlook() calls and their aftermath. The fix involved a rather extensive rewrite of mlook() and mxlook(), probably negating a good deal of the performance improvements gained in the tuning extravaganza of last year, but I also added an early loop exit that, hopefully, negates the negation. The various demos and torture test still work. ckuus5.c, 2 May 2001. Straightened out messy if-condition in doxget() to complain about missing as-name for XMODEM receives. ckuus6.c, 2 May 2001. Cleaned up most gcc -Wall warnings in ckuus7.c, including (don't ask me why) putting braces around all keyword tab entries. Ditto for ckuus4-6, but I doubt I'll have the patience to do this for all the modules. 2 May 2001. Fixed the ECHO command (again) so that it works as before when given a quoted string, so that echo "foo" will print "foo" (with the quotes). The previous fix was a bit flaky, e.g. echo foo "bar" printed: "foo "bar"" But even with the bugs fixed, I'm still not sure whether this is the right thing to do. It keeps the ECHO command working as it did before, but it also will confuse the heck out of somebody who expects the doublequotes to be stripped. But it really doesn't seem worth adding a SET command for this... ckuusr.c, 3 May 2001. Lint/Wall cleanups, cont'd. ckuus[rxy].c, ckcftp.c, 3 May 2001. The trick for looking ahead to see if we're at the end of a macro definition (used by CONNECT /ASYNC) is not reliable because sometimes macro definitions have trailing spaces and/or commas. These can be added by the parser when reading a multiline { block } from a file, the prompt, or even another macro. We can (a) prevent the trailing ", " from being added in the first place (but this would have to be handled in at least three places); or (b) change the DEFINE command to remove it; or (c) change the CONNECT /ASYNC lookahead code to test for only commas and spaces left in a macro definition. (c) is obviously the safest choice since it can't possibly break anything, but it's ugly and unsatisfying. The problem with (a) is that it can't address nested definitions; (b) is out of the question because how do we know it's a macro and not a variable whose definition is *supposed* to have trailing comma and/or spaces? Which leaves (c). I added a new routine, prepop(), to replace the loop which had been replicated in various places around the ckuus*.c files, which does what the loop did but also checks whether nothing remains in a macro definition except spaces and/or commas (this is done by a second new routine, iseom()). ckuusr.h, ckuus[r57].c, 4 May 2001. Feature-set builds on Linux, 4 May 2001: Build Size Savings Corrections needed Full 1975235 - None NOSPL 1659767 315468 ckcmai.c ckuus5.c ckcftp.c ckudia.c NOCSETS 1597924 377311 None NOUNICODE 1682029 293206 None NONET 1704588 270647 None NOFTP 1834813 140422 None NODEBUG 1761288 213947 None NOHELP 1692150 283085 None NOLOCAL 1442034 533201 ckudia.c NODIAL 1834898 140337 ckudia.c MINIDIAL 1945894 29341 None NOCHANNELIO 1943741 31494 None NOCMDL 1932120 43115 ckcmai.c NOIKSD 1925070 50165 None NOSEXP 1953885 21350 None NOFRILLS 1791245 183990 ckuus2.c NOCURSES 1946989 28246 None NOXFER 1445466 529769 None NOICP 476762 1498473 ckcmai.c ckuus[r3457y].c ckcftp.c ckcdeb.h NOSERVER 1946942 28293 None NOPUSH 1930325 44910 ckcftp.c Notes: . NOICP for now implies NOFTP, although a command-line-only FTP client could be accomplished with a lot of #ifdef fiddling . NOXFER also implies NOFTP. Built non-ANSI versions on SunOS and HP-UX; no changes needed. 4 May 2001. Built on VMS with & without TCP/IP, required a fix to ckvcon.c. 4 May 2001. Fixed REDIRECT to require an arg. ckuusr.c, 4 May 2001. Enabled SET SESSION-LOG for VMS and other platforms where it wasn't usable before, no now they can get timestamped session logs. ckuus[r3x].c, ckvcon.c, Cinco de Mayo 2001. Command recall didn't work well when reversing direction; for example, after up up up, down would show the same command again, similarly in the other direction. Fixed in gtword(), 5 May 2001. SET FLOW /xxx AUTO should not have been allowed. AUTO is not a valid choice for a specific connection type. Fixed in ckuus3.c, 5 May 2001. The SHOW FLOW display was confusing about AUTO versus specific types. Fixed in ckuus4.c, 5 May 2001. IKSD URLs (added 23 Dec 2000) were broken at some point. Fixed in prescan(): ckuus4.c, 5 May 2001. Checking verbosity of FTP commands... . PUT/GET always prints the file size. . If a mode change is done, that is printed too. Example: (/w/fdc/tmp3/) C-Kermit>get foo.txt 46669 Type set to A. . If TRANSFER DISPLAY is BRIEF, we still get a lot of stuff: (/w/fdc/tmp3/) C-Kermit>get foo.txt 46669 Type set to A. GET foo.txt (text) (46669 bytes)Entering Passive Mode (128,59,39,2,15,154) Opening ASCII mode data connection for foo.txt (46669 bytes). Transfer complete. : OK (0.196 sec, 238107 cps) (/w/fdc/tmp3/) C-Kermit> . Debugging display shows none of this comes out with DISPLAY FULL. Going through all the FTP commands with VERBOSE ON, which is the default, shows they are very inconsistent about showing server responses. Let's think about this.... We have three kinds of messages: 1. Messages from Kermit to the server, which are currently shown only if FTP DEBUG is ON. 2. Messages from the server to Kermit, which are shown inconsistently, but when they are shown at all, FTP VERBOSE must be ON, which it is by default, producing annoying messages in the DIR, VDIR, CHECK, DELETE, and MODTIME commands, as well as the file-transfer commands. 3. Messages from Kermit to the user, which are shown unless QUIET is ON. We went through all this a month ago but didn't settle it. OK, here's what I did: first I made FTP VERBOSE OFF be the default (again), because it produces the best results for all commands except BYE, STATUS, and PWD; i.e. the ones where the server sends back data on the control connection. For these we use the "printlines" variable to force verbose mode unless QUIET is ON. ckcftp.c, 8 May 2001. Peter E complained about the well-known (to me) situation in which you give a valid command, then some more commands, then ^P back to the previously valid command, but it's not valid any more (e.g. because it deleted or renamed a file, so the file no longer exists, so cmifi() gets a parse error). This one has been on my list for a long time because it's so hard to fix. Command recall is done in gtword(). When ^P or ^N is encountered, the previous or next command is copied from the history list and stuffed into the command buffer, and then gtword() returns with cmflgs = -1, which forces a reparse, and this is what causes the error. But what if, instead of returning, gtword() simply continued its character-getting loop until it gets a CR, DEL, SP, or other break character is entered? Sounds good in theory, and I almost got it working, but it has too many side effects -- you have to type two CR's to enter a recalled command; recalled commands don't erase longer previous commands so there is junk on the end; ?-help is totally confused; the program sometimes just hangs, etc. So rather than try to fix each of these side effects piecemeal by adding more and more but-ifs, I backed off. 8 May 2001. Peter E noticed that DELETE did not print an error message if it failed to delete 1 or more files and /NOLIST was in effect (as it is by default). Added a special case for this to dodel(): ckuus6.c, 9 May 2001. Updated NEWS text plus a few assorted HELP-text glitches. ckuus2.c, 9 May 2001. Fixed FTP VERBOSE vs FTP [M]GET and FTP DELETE -- there was still some interference. ckcftp.c, 9 May 2001. More FTP message fixing from Jeff (for authentication of data channel). ckcftp.c, 10 May 2001. Change SINIX 5.42 makefile entry to not optimize -- otherwise it gets stuck on ckuusr.c, ckuus4.c, etc. 10 May 2001. ---Beta.01--- 22 May 2001: I found a bunch of problems with FTP and Jeff worked on them: > With C-Kermit or K95 (latest), CD orientation messages aren't printed. > Another VERBOSE mode foulup. What a nightmare. But that's only the > beginning. ok. this I can fix. > If I SET FTP VERBOSE ON (but not DEBUG) and then CD to a directory, the CD > messages are printed, but the beginning of each line is lost. There's some > new bug in getreply(). I know where this would be > I wanted to get a partial binary file so I could REGET it afterwards. "get > wermit" (from K95 to watsun) gets wermit in text mode. That's bad but I can > understand it: automatic text/binary switching is based on filetypes, and > this one doesn't have a file type, and the platforms are not alike, so it > didn't switch to binary mode -- but why is the default text mode???. > > But "get /binary wermit" still gets it in text mode! The /BINARY switch > should override everything else! > default is TEXT because the default for FTP is TEXT. It was probably never changed. Not sure why /BINARY does not override. But it is probably because the FTP TYPE value is copied over the current value at some inappropriate point. > "ftp type binary" gets around this but... "get wermit" and then interrupt > with Ctrl-X (seems OK). Then "reget wermit" goes into an uninterruptible > infinite loop of "ftp: connect: No such file or directory" and "Transfer > interrupted". I had to Ctrl-Alt-Del it. > What does SET FTP DEBUG show I wonder .... > When I started a new copy of K95 and make an FTP connection to ftp.kermit, > log in anonymously, cd kermit/bin, "get msvibm.zip", interrupt it with 'X' > (so far so good), and then "reget msvibm.zip", I get: > > ************************* > RECEIVE- or GET-class command failed. > Packets received: 21505 > Damaged packets: 0 > Timeouts: 0 > Packet length: 4000 > Most recent local error: "Bad file descriptor" > ************************* > > That's a Kermit message. The FTP connection was closed. That's apparently > what happened the first time too? I wonder where K95 thought it was > REGET'ing the file from... Interesting ... > Starting a new copy of K95, doing the same thing again with a debug log, the > same thing happens, except now the File Type: and File Size: fields in the > file transfer display are blank??? So the debug.log is causing the file type and size to not be set? > At this point (after 'X') there is no connection (verified by SHOW COMM and > SHOW CONN). > > So then I told the same copy of cknker to "ftp watsun" so I could upload the > debug log. It put up the "Name:" popup, but typing goes to the prompt and > not to the popup. The popup sits there uselessly and there's no way to make > it go away. I had to exit again and start a new copy. > > Now I make an ftp connection ftp.kermit, logged in anonymously, set ftp > debug and verbose on, file display brief. Then "rcd kermit/bin" and this > time I got the CD orientation messages correctly, with no missing pieces. > So apparently this works only when FTP DEBUG is ON. > > Now I typed "reget msvibm.zip" (because I still have a partial msvibm.zip > in my current directory) and K95 said "Sorry, GET /RECOVER requires > binary mode." I thought I put code in months ago to force binary mode for > REGET!!! > > OK, so "ftp type bin", "reget msvibm.zip". Now we get a neverending series > of: > > 501 Invalid number of arguments. > GET (binary) (-1 bytes)---> PASV > 227 Entering Passive Mode (128,59,31,95,8,133). > ftp: connect: Bad file descriptor > : INTERRUPTED > ---> SIZE > 501 Invalid number of arguments. > GET (binary) (-1 bytes)---> PASV > 227 Entering Passive Mode (128,59,31,95,8,134). > ftp: connect: Bad file descriptor > : INTERRUPTED > ---> SIZE > 501 Invalid number of arguments. > GET (binary) (-1 bytes)---> PASV > > which can't be interrupted. Well, Ctrl-C is caught (because it echoes > "^C..." but it doesn't stop anything. I have to Ctrl-Alt-Del it. > > Now I switch to UNIX. I used C-Kermit to make an FTP connection to watsun, > interrupted a GET with 'x' and it seemed to work: > > C-Kermit>get wermit > ---> SIZE wermit > 213 2244608 > GET wermit (binary) (2244608 bytes)---> PASV > 227 Entering Passive Mode (128,59,39,2,7,35) > ---> RETR wermit > 150 Opening BINARY mode data connection for wermit (2244608 bytes). > 426 Transfer aborted. Data connection closed. > 226 Abort successful > : INTERRUPTED > C-Kermit> > > SHOW CONN says the connection is still open. A second "get wermit" works, > and can be interrupted successfully. And a third, and so on. But: > > C-Kermit>reget wermit > ---> SIZE > 500 'SIZE': command not understood. > GET (binary) (-1 bytes)---> PASV > 227 Entering Passive Mode (128,59,39,2,7,242) > ---> RETR > 500 'RETR': command not understood. > ---> SIZE > 500 'SIZE': command not understood. > GET (binary) (-1 bytes)---> PASV > 227 Entering Passive Mode (128,59,39,2,7,244) > ---> RETR > 500 'RETR': command not understood. > > and so on forever. Obviously we're sending SIZE and RETR commands without > operands -- something has been lost. But then why does it loop? > > I think most of the problems have been fixed (if they can be): /BINARY switch now does the correct thing. The reason it was not working is that the PATTERNS code in getfile() did not pay attention to the 'forcetype' flag which is set when a switch /BINARY /TEXT ... is used. The reason that ASCII transfers are the default is because according to the FTP protocol specs, ASCII transfers are the default. File Collision Options were a bit weird. The first time you issued an FTP command the current File Collision Option would be assigned to FTP File Collision Option IFF the SET FTP COLLISION command had never been issued. But if you then issued a SET FILE COLLISION command the original one would still be active for FTP. Very confusing since there was no way to determine what the FTP Collision setting would be. I changed this to behave as follows: if the SET FTP COLLISION command is not executed, the value of the SET FILE COLLISION command is used. The reason there was a problem with REGET going into an infinite loop is that the 'getone' flag was not being set. Therefore, if there was an error when calling ftpgetfile() the program would go into an infinite loop attempting to REGET the same file over and over again. Only problem was the second time through the loop the same IP-Addr/Port combination would be used. Unfortunately, since the initial REGET failed the host was no longer waiting for a data connection. The connect() would fail and the control channel would be closed as well. Fixed by setting 'getone' for REGET as well as GET. FTP REGET of files now work if the appropriate File Collision settings are used. REGET with UPDATE does not work if the Timestamps do not match. BINARY is not forced for REGET. If you say FTP TYPE TEXT FTP GET file and then FTP REGET file it is absolutely correct for the REGET to be rejected because the previous transfer type was TEXT. However, FTP REGET /BINARY file will force the transfer to be performed anyway. I tested this and it works. The problem with initial banners not being displayed was caused by an incorrect setting of the verbose parameter for some of the many "PASS" command instances. The problem with the first four characters being deleted was caused by an incorrect test in ftp_reply(). This is the code that was added to strip the numeric responses. However, this code was never written to take into account the fact that some responses are multiple lines and the numeric response only occurs on the first line. I can't seem to find an ftp server that sends the directory README file. But I think the same change that fixed the above code would work there. File Transfer display Size and Filename. This is a problem with SSL/TLS connections that I do not believe can be solved in a trivial manner. It has to do with the code in ssl_dataconn() that turns off the file transfer display because of the potential user prompts which can occur if the server certificate info does not match what the user specified when making the connection. There was a memory corruption error if /USER: and /PASSWORD: were used on the FTP OPEN ... command line. Memory was being free'd but the pointers were not set to NULL. Subsequent commands would act in a random manner. ^C is not processed in the ftp code in K95. None of the FTP uses the hokey ^C processing used in the command parser. (End of quoted email from 22-23 May 2001) Glen Thobe noticed that in Solaris 8, if you make an FTP connection, GET a file, and then try to GET a file that doesn't exist, C-Kermit loses its terminal modes and no longer echoes commands. This seems to be a double-barreled problem. First of all, doftpget() did not call ftscreen(SCR_TC,...) if the operation failed, so of course the echoing was messed up after a failed FTP GET. (Yet oddly enough, this showed up only in Solaris 8...) But fixing this didn't cure the problem. Why? Because the SCR_TC code in screenc() contained this: #ifndef VMSCURSE endwin(); #ifdef SOLARIS conres(); #endif /* SOLARIS */ #endif /* VMSCURSE */ Unfortunately there is no explanatory comment, nor anything in my notes about why/how/when the SOLARIS bit was added. But taking out the call to conres() fixes the problem and doesn't seem to break anything in Solaris 2.5.1. ckcftp.c, ckuusx.c, 23 May 2001. From Jeff: a fix for date-time strings sometimes having two extra 0's on the end, e.g. in DIR listings. ckuus6.c, 24 May 2001. From Jeff: fix SHOW PROTO to print auto-blah strings with shostrdef() so control chars appear as (e.g.) \{13}. ckuus4.c, 24 May 2001. From Jeff: fix FTP PWD, CD, CDUP to always display replies (including CD orientation messages); fix a misplaced brace in ftpopen(). ckcftp.c, 24 May 2001. Define BIGBUFOK for all SOLARIS, not just Sparc. ckcdeb.h, 24 May 2001. Set NEWDEFAULTS if BIGBUFOK is defined. Previously NEWDEFAULTS was set only if OS2 was defined. ckcker.h, 24 May 2001. Made SET FTP DA be an OK abbreviation for DATES. ckcftp.c, 24 May 2001. Kermit server was not discarding an incompletely received file if the retry limit is exceeded while trying to read an ACK. In this case it calls errpkt() to send "Too many retries" and errpkt() calls clsof(). But (a) it was calling clsof() with the wrong argument (discard) and (b) clsof() was being called again later. Fixed errpkt() to call clsof(1) and then let clsof decide about discarding or keeping (which it already did), and the protocol module to skip calling clsof() a second time if it knew it has already sent an error packet. ckcfn2.c, ckcpro.w, 24 May 2001. It is increasingly likely that our fixed stringspace allocation will be used up when building big and/or recursive file lists. For BIGBUFOK UNIX builds, the pool is currently set at 500000. If it runs out on any particular operation, it simply gets an error. There's no way to make it handle more filenames. Of course we could always just make the pool bigger, but that penalizes everybody. Or does it? With 0.5MB filename stringspace (as in 7.0), Kermit uses about 2700K when doing a directory listing of a directory with 2300 files. When I double the stringspace to 1MB, it still uses 2700K. But when I raise it to 5MB, it still uses 2700K. This suggests that it might be OK to simply make it bigger (because on most platforms, the memory is not actually allocated until used), rather than coding some complicated dynamic extension scheme. But how big? No number will satisfy everybody so we'll need a command to change and reallocate. Same deal for the maximum number of filenames, currently 102400 (= 400KB) for BIGBUFOK builds. So I left string space at 0.5MB for UNIX BIGBUFOK builds (since, after all, only one person has ever complained about it in all these years). This allows 100,000 files with an average namelength of 5 characters, or 50,000 files with an average name length of 10, etc. Then for adjustments in the #ifdef UNIX / #ifdef DYNAMIC case, I added SET FILE STRINGSPACE and SET FILE LISTSIZE to let the user to change the stringspace and the maximum number of files. Added this info to SHOW FILE also. ckcker.h, ckuus[247].c, ckufio.c, 24 May 2001. Now that we can change these two allocations independently, it's easy to stress-test the code. Sure enough, I found that UNIX C-Kermit would dump core if the filename pointer array filled up before the string space did. Added a test for this. Also ensured that if a too-large operand results in a malloc failure, the old space remains undisturbed and usable. ckufio.c, 24 May 2001. Solaris 8 serial ports and dialing on PC, built with gcc 2.95.3... . Lockfile is SVR4 style with device numbers (OK) . /dev/term/a and /dev/ttya interlock properly. . But /dev/term/a and /dev/cua/a do not interlock. . However, if /dev/term/a is open then /dev/cua/a gets "device busy". . And vice versa. . Meanwhile, there is a secondary lockfile with a bogus name "fU". I wonder where this comes from. The file 'a' in /dev/term is a symlink: a -> ../../devices/isa/asy@1,3f8:a The bogus secondary lockfile name was because of a misplaced #ifdefs. But when we use device numbers in the lockfile name, there is no need for a secondary lockfile (which is used only to account for multiple names for the same device). Fixed in ckutio.c, 24 May 2001. . Lockfiles are correctly deleted upon closing the device. . Stale lockfiles are correctly removed. Uploading a big file at 115200 bps (on a V.90 connection) works fine (the Solaris header files define higher rates but the port device driver doesn't accept them). RTS/CTS is fully effective. Protocol defaults are now modern (big buffers, etc). Echoing of commands after curses display works fine. Suspending and continuing are OK, including terminal modes. Shell terminal modes are normal after quitting. In short, I couldn't reproduce any of problems that were reported recently. Of course this is on a PC, but the complaints came from Sparc users. Add -DNOFLOAT to Coherent 4.2 makefile entry. makefile, 1 Jun 2001. Lars Kellogg-Stedman noticed that the PTY command wasn't working in recent Linuxes. Rug-pulling-out syndrome again. Fixing it broke it for old Linuxes like Red Hat 5.2 (surprise). The most foolproof fix seemed to be to rewrite the makefile entry to test for the presence of /dev/ptmx, with corresponding shuffling of #ifdefs in the pty module: separation of HAVE_STREAMS from HAVE_PTMX (needs checking on non-Linuxes, but I think only SINIX used HAVE_STREAMS...) Experimental for now. One good byproduct of this is that the new linux makefile entry (xlinux) is not only simplified but much more flexible -- almost a configure script in itself, but without the previous deeply nested if-then-else's, so new tests can be easily added. ckupty.c, makefile, 5 Jun 2001. After additional testing and verification of new Linux entry, removed the old one and made the new the "make linux" target. Made sure the ptys work on Solaris, SunOS, Linux, and SINIX. makefile, 6 Jun 2001. From Jeff, 7 Jun 2001: . New makefile targets: solaris2xg+openssl+zlib+pam+shadow solaris8g+openssl+zlib+pam+shadow solaris8g+krb4 . ckupper() added to ckclib.[ch], like cklower(). . Fix to a debug() statement for packet CRC, ckcfn2.c. . Change SET TELNET COMPORT to COM-PORT to match RFC spelling, ckuus[r3].c. . Move misplaced paren in dump[rs]buf() printf(): ckcfn3.c. . New ckuath.c, ckctel.h, ckcnet.c, ck_ssl.c (didn't look at them). Jeff also changed the -d command-line option to produce a timestamped debug log, probably temporarily and forgot to put it back, but since there is no way to do this from the command line, I added a quick hack to turn on timestamping if -d is included more than once, in the spirit of many other UNIX programs that select the debugging level by the number -d's. prescan(): ckuus4.c, 7 Jun 2001. Made -DSOLARIS8 imply -DSOLARIS7, which, in turn, implies 2.6, 2.5, 2.4, ... (in other words, added the missing "each Solaris version implies all earlier ones" clause for Solaris 8). Added the various SOLARISxx version symbols to SHOW FEATURES (back to 2.4). ckcdeb.h, ckuus5.c, 7 Jun 2001. Simplified Solaris makefile targets by removing -Dxxx's that are now taken care of ckcdeb.h. makefile, 7 Jun 2001. From Lucas Hart: Addition of machine-model info for OSF/1 and Solaris. Changed the governing #ifdefs to be conservative about which versions get this code (Tru64 UNIX only, not all OSF/1's, and Solaris 2.5 and later) since we can't verify on earlier versions. Used by debug log, \v(model), and SHOW FEATURES. sysinit(): ckutio.c, 7 Jun 2001. From Lucas: Fix an off-by one error in setting the architecture name in the VMS version (VA instead of VAX, etc). ckvtio.c, 7 Jun 2001. From Jeff, 24 Jun 2001: . Add #include to pick up ... ckcnet.h . Fix a null-pointer-dereferencing vulnerability in zzstring(): ckuus4.c. . Add prototype for ckupper(): ckclib.h. . Add FD_SETSIZE display to SHOW FEATURES: ckuus5.c. . Fill in a missing IKSD REMOTE LOGIN case: ckcpro.w. . Fix an HTTP proxy host identification detail in netopen(): ckcnet.c. . Fix a pair of potential 1-byte memory leaks: ckctel.c. . Fix FTP OPEN to expand variables in username, password, & acct: ckcftp.c. . New ck_crp.c and ckuath.c modules. . Updated and new security targets + new, secure RH7.1 target: makefile. Jeff reported a problem in which "def xx ask \%c prompt:" would not display the prompt when xx is executed, which I can't reproduce in C-Kermit or K95. But in fooling with this I noticed that you can't type ? in the response text any more. It worked in 6.0 and has been broken since 7.0. Fixed in gtword(), which needed a special case for '?' in ASK. ckucmd.c, 24 Jun 2001. A related but more difficult problem is when a command contains a '?' because it was typed with a preceding '\' to suppress its normal help function, it goes into the command buffer with the '\' stripped; thus when you recall the command with Ctrl-P gtword() thinks you typed '?' for help. gtword() already had a flag, chsrc, that tells whether the current character came from the keyboard, so I just needed to test it before entering the code that processes ?, Esc, Tab, Rubout, ^U, etc. ckucmd.c, 24 Jun 2001. There was a related problem with files that have question marks in their names. Suppose a file is named "abc?123". If you want Kermit to TYPE it, you can't use "type abc?123" because that gives help. If you use "type abc\123", this forces the command parser to accept the question mark, but when cmifi() gets result, it thinks it's a wildcard rather than a literal character. So it must be quoted for cmifi() too: "type abc\\\?123". This command works (but it didn't work in 6.0 and 7.0), and now recalling it works too. One last item in this area: also skip the ?/Esc/Tab/Rub/^U/etc processing if stdin is not a tty. gtword(): ckucmd.c, 24 Jun 2001. Jeff said "mget makefile ck[cu_]*.[cwh]" did not work between K95 and CK, but it works for me, no matter which side is the client and which is the server. All the files are sent correctly and nobody says "file not found". Dell Coleman mentioned recently that SET FILE DOWNLOAD-DIRECTORY did not work for XYZMODEM. There's no reason why it shouldn't. Added code for this to the non-Kermit section of xxproto(): ckcpro.w, 24 Jun 2001. We already have a GREP command and a /COUNT option, but no way to access the result programmatically, so I added an optional variable-name argument to /COUNT, e.g. "grep /count:\%x XYZ ckc*.c". ckuus[26].c, 24 Jun 2001. Changed dumprbuf() and dumpsbuf() to avoid warnings on machines with 64-bit pointers; needs testing on Tru64 5.0A (which I don't have access to): ckcfn3.c. Commented out some debug() statements in ckuxla.c for the same reason. It's pretty much no longer ok to assign addresses to ints or even to longs, or to print them, without taking special pains. 24 Jun 2001. Gerry Belanger wanted an idle limit in C-Kermit, like there is in K95. It's easy enough to add in the select() versions as SET TERMINAL IDLE-LIMIT (rather than CONNECT /IDLE-LIMIT:n, which is too tied up with #ifdef OS2's). SET TERMINAL IDLE-LIMIT should now also be effective in K95. ckuusr.h, ckuus[27].c, ckucns.c, 24 Jun 2001. But what should happen when the idle-limit goes off? In k95, it simply returns to command mode. There is no indication of *why* it returned to command mode, so how can a script know what to do? Added a new variable: \v(cx_status). Values are listed in ckcker.h as CSX_xxx, and are set in ckucns.c in every spot where it returns. We can mess with the values for now, but they have to be stable before we release, since scripts will depend on the actual numeric values. To add to K95, search ckucns.c for all occurrences of "cx_status" and add the corresponding statements in the K95 modules; ditto for VMS, etc (still to be done). For docs, note that idle-limit is enforced only while CONNECT mode is active; not in command mode, and not when PUSH'd from CONNECT mode, not while executing APCs, not while running scripts, etc. Maybe this could be done in K95 (which has threads), but not in C-Kermit. ckcker.h, ckuus4.c, ckucns.c, 24 Jun 2001. This is fine for scripts, but what if we want the connection to hang up automatically after an idle timeout without making the user run a script? (Which is what Gerry wanted in the first place.) What we really need is a way to specify the idle timeout and the timeout action separately: return to command mode but keep the connection open, send a string and remain in CONNECT mode, hangup and return to command mode, or whatever else we can think of. From Jeff: new CSX_blah (CONNECT status) values for K95 (but I wish you would use shorter symbols in the universal header files -- some old compilers might choke on these -- nothing is gained by making them so long). From me: IDLE_blah symbols for idle actions. ckcker.h, 25 Jun 2001. Added: SET TERMINAL IDLE-TIMEOUT SET TERMINAL IDLE-ACTION { HANGUP, OUTPUT , RETURN, EXIT, ... } This supersedes SET TERM IDLE-SEND (which still works but is invisible). The idle timeout is now tt_idlelimit (secs), but SET TERM IDLE-STRING also still sets tt_idlesnd_tmo in case anybody still refers to it. SET TERM IDLE-ACTION sets tt_idleact; the values are defined in ckuusr.h as IDLE_blah symbols. Changed SHOW TERMINAL to show new TERMINAL IDLE-blah settings. ckuusr.[ch], ckuus7.c, 25 Jun 2001. Changed UNIX select() CONNECT module to follow the new TERMINAL IDLE-ACTION and IDLE-TIMEOUT settings. See CKTIDLE sections. ckucns.c, 25 Jun 2001. Removed TERMINAL IDLE-SEND from the HELP text and added help text for the two new options. Also removed the clause that says that variables and Kverbs are evaluated at transmission time, because this was never true (the cmtxt() call that parses the string passes it through xxstring()). ckuus2.c, 25 Jun 2001. Uppercased HELP, INTRO, LICENSE, and SUPPORT keywords in the top-level command table so they stand out better in top-level ?-help. Made the following top level commands invisible, since they are confusing and/or rarely used and/or "deprecated" and/or useless at top level: ASSERT, EIGHTBIT, IKSD, LOCAL, LS, QUERY, and REINPUT (IKSD is confusing because, if you didn't know what it did, it could just as easily start an IKSD server as make a connection to one; I can always make it visible again, but maybe we need a less confusing name for telnet'ing to an IKSD server.) ckuusr.c, 25 Jun 2001. Added an SSHCMD symbol to ckuusr.h, which is defined if NETPTY is defined and NOPUSH is not defined, or if SSH is defined. 25 Jun 2001. In the NETPTY case only, added SSH and SET SSH COMMAND commands, which appear if SSHCMD is defined. The default SSH command is "ssh -e none". The SET SSH COMMAND value is shown by SHOW NET. Also filled in HELP SSH and HELP SET SSH. This makes SSH visible, and creates a transparent connection by default so we can transfer file through it (normally its esape character is tilde, like rlogin and cu, which would clobber the very first Kermit packet). ckuusr.[ch], ckuus[2347].c, ckucns.c. 25 Jun 2001. Note: on watsun, the SSH command usually works, but sometimes fails with "/dev/ttyp6: permission denied" -- I don't recall seeing this before, but I didn't touch the pty module or ttopen()... Removed the telephone section from the SUPPORT command text. Everybody should be able to handle email by now. ckuus6.c, 25 Jun 2001. Fixed out-of-order keywords in TELNET-command switch table. ckuus7.c, 25 Jun 2001. Added a second argument to gtword() for options, changed all calls to gtword() to supply 0 for this arg. Then tried defining an option to disable ?-help and \-quoting in fields that were in doublequotes, but quickly got into big trouble so backed off. ckucmd.c (no changes), 25 Jun 2001. Since we're going to have SSH, and because more time has passed, and for extra hype, changed the C-Kermit version from 7.1.199 to 8.0.200. Still need to change all the module version numbers... ckcmai.c, 25 Jun 2001. From Jeff: shorten new CSX_xxx symbols in ckcker.h; put \v(cx_status) in #ifdef NOLOCAL in ckuus4.c, 26 Jun 2001. Added SET ATTRIBUTE FORMAT { ON, OFF } to let C-Kermit adapt to Tandem Kermit that sends a whacky record-format code. ckcmai.c, ckuus7.c, 26 Jun 2001. Considered adding a new protocol option to negotiate UTF-8 filenames, independent of the file and transfer character set; a great idea in principal, but a BIG can of worms in practice. So I tabled the idea indefinitely (writeup available on request). In any case, at least I worked out a safe way to add more capability bits. 26 Jun 2001. Fix from Jeff for HTTP proxy code. ckuus7.c, ckcnet.c, ckcftp.c, 27 Jun 2001. New SSL targets for Solaris 2.6 from Joerg Heitkoetter. Makefile, 27 Jun 2001. Changed version strings in all modules from 7.1.xxx to 8.0.xxx. 27 Jun 2001. Set cx_status in VMS CONNECT module. ckvcon.c, 27 Jun 2001. Moved shostrdef() prototype from ckuus7.c to to ckuusr.h, and added casts to non-(CHAR *) shostrdef() args in ckuus4.c, 27 Jun 2001. Added missing \n\ at end of continued string constant in HELP SSH text. ckuus2.c, 27 Jun 2001. Added missing #ifdef CKTIDLE..#endif around getiact(). ckuus7.c, 27 Jun 2001. Fixed Ctrl-\u in VMS CONNECT on Telnet connection -- it didn't do anything, now it closes the connection. ckvcon.c, 27 Jun 2001. Discovered that the FAST and CAUTIOUS (but not ROBUST) commands were commented out. My notes show that this happened 25 Mar 1999 but then I thought I undid it a week later, but evidently I only undid for ROBUST and not the other two. Now they are uncommented but invisible, which was the original intention. Strange that nobody noticed this in 2+ years... ckuusr.c, 27 Jun 2001. Mark Sapiro reported the FreeBSD 4.x echoing problem had resurfaced in FBSD 4.0 in Beta.01. Unfortunately I only have access to 3.3 and 4.3. It doesn't happen on either of those... SSH on FreeBSD 4.3 localhost connection: interactive connection works, scripts work, but streaming file transfers just sit there in the first data packet. Turning streaming off fixes it. It also works with streaming on, but only when the packet size is 1K or less. The same thing happens with a long-distance SSH connection, so evidently it's a problem with the FreeBSD pty. However, there is no problem RECEIVing with streaming on an SSH connection (and we get pretty good speed too, like 700Kbps). Still on FreeBSD... Discovered that neither it nor any of the other *BSDs included Rlogin support, not that it matters much since you can't use it unless you're root. Added BSD44 to the list of RLOGCODE platforms in ckcnet.h (adds about 7K to the binary), 27 Jun 2001. UnixWare 7.1.1: SSH file transfers work fine, streaming and all. Discovered that fullscreen file transfer display, when it prints the "Network type:" value, indexes into the netname[] array without checking bounds. This array was not updated when NET_PTY was added as a network type, so SSH or any other pty-based connection caused a wild memory reference. Added code to check the bounds, plus some dummy entries that will show up automatically the next time we add a network type without updating this array. Also, the netname[] array contained a bogus "SSH" entry, which threw everything after it off by one. ckcnet.h does not define a network type of SSH. ckuusx.c, 27 Jun 2001. Corrected spelling "psuedoterminal" -> "pseudoterminal" in several places. ckuus4.c, 27 Jun 2001. Built without incident on HP-UX, which doesn't have an ssh client installed. "ssh watsol" says: /dev/pty/ttyp3: Permission denied ?Initialization failure which I don't understand; "pty ssh watsol" does something *slightly* more reasonable: enters CONNECT mode and then immediately pops back to the prompt (no error message). "pty telnet watsol", however, works fine. Hmmmm, but after doing these, "ssh watsol" doesn't get the "permission denied" error any more and behaves just like "pty ssh watsol". The same thing was happening on watsun yesterday. I wonder what the deal is... Anyway, I suppose I could try to prevalidate the "ssh" command by doing a PATH search or somesuch, but that does not take into account the user's shell aliases and so forth. So unfortunately there's no good way to give a meaningful error message if the 'ssh' command is not found. Mark Sapiro reports FreeBSD 4.0 OK after all; he used the wrong makefile target because the makefile didn't list the freebsd41,42,43 targets in the comments at top. Fixed in makefile, 28 Jun 2001. Changed logpkt() to convert the packet-start character to Caret-Letter notation so looking at packet logs in the Kermit terminal screen does not trigger an autodownload. ckcfn2.c, 28 Jun 2001. HELP SET TERM didn't include IDLE-TIMEOUT. ckuus2.c, 28 Jun 2001. Feature-set builds on Linux, gcc 2.96, 28 Jun 2001: Build Size Savings Corrections needed Full 1984671 - None NOSPL 1667990 316681 ckucmd.c ckuus6.c NOCSETS 1607616 377055 none NONET 1710743 273928 ckuusx.c NOFTP 1843513 141158 none NODEBUG 1769572 215099 none NOHELP 1699954 284717 none NOLOCAL 1445894 538777 ckuusr.h NODIAL 1844366 140305 none NOXFER 1452623 532048 none NOICP 479462 1505209 none NOPUSH 1937601 47070 none SSH funnies... . /dev/ptyxxx: Permission denied. Seems to happen just about everywhere, but if you try again, it doesn't happen. . SSH connection to HP-UX 9.05 -- logging out from HP-UX doesn't close the connection (but others OK...) . File transfer works mostly fine. We get a few errors/retransmissions but that's OK because Kermit does not consider this a reliable connection and so does not stream. (But AIX gets LOTS of errors unless a small packet size, like 800, is used.) . SSH doesn't use net directory... . Xfer display Network Type: says TCP/IP instead of PTY... Added notes about secure entries to makefile. 28 Jun 2001. Linux... The new makefile entry works pretty well (everywhere but Debian 2.1). Changes that didn't get into all the binaries, 29 Jun 2001: . zchin() return code correction for ZIFILE, ckufio.c. . Corrections to HTTP command error handling, ckcnet.c. . Print error message on pty allocation failure, ckupty.c. ---C-Kermit 8.0.200 Beta.02--- Added sco32v505x and sco32v505xnet entries that work on my SCO OSR5.0.5 installation. makefile, 29 Jun 2001. Added linuxnp entry for Linuxes that have /dev/ptmx but do not have grantpt(), unlockpt(), or ptsname(), such as Debian 2.1. makefile, 29 Jun 2001. Fixed typos reported in SCO OSR5.0.5 gcc targets reported by Emilio Perea. makefile, 30 Jun 2001. Added -DNETPTY to Ultrix 4.4 and 4.5 entries and put #include within #ifndef ultrix..#endif in ckupty.h, so the Ultrix version can be compiled with PTY support. But it doesn't work at runtime. The base Ultrix 4.x entry had -DNODEBUG, so removed that and rebuilt to debug (which went OK despite warning about "bizarre relocation errors" in the makefile comments): pty_getpty() found pty master[/dev/ptyp1] pty_getpty() slavebuf [2][/dev/ttyp1] do_pty() Xline[/dev/ttyp1] do_pty()[Slave starts] getptyslave() ptyint_void_association()[setsid()] ptyint_void_association() open(/dev/tty,O_RDWR)[/dev/tty]=-1 Here we try to get a handle on /dev/tty so we can do a TIOCNOTTY ioctl on it to void the association with the console, but the open() fails (why?) But according to the code, this is not fatal... pty_open_ctty() slave[/dev/ttyp1] ptyint_void_association()[setsid()] ptyint_void_association() open(/dev/tty,O_RDWR)[/dev/tty]=-1 <-- same deal pty_open_ctty() open ok[/dev/ttyp1] pty_initialize_slave()[fd]=7 pty_initialize_slave()[pid]=3575 pty_open_slave() open failed[/dev/tty] getptyslave()[Unable to open slave]=44806918 do_pty()[getptyslave() fails - exiting] do_pty()[Slave fails to initialize] Another failure to open /dev/tty. A peek at the code reveals that all these failures are in #ifndef NO_DEVTTY sections... Sure enough, defining NO_DEVTTY removes those sections and makes it work. makefile, ckupty.c, 30 Jun 2001. (But later reports indicate that the Ultrix pty driver is not well buffered and flow-controlled and can't handle file transfer thru ssh.) Minor adjustments to the new ultrix44 and 45 makefile targets (which worked with DEC make) for GNU make. 1 Jul 2001. Updated Coherent 4.2 makefile target from Fred Smith. makefile, 2 Jul 2001. Put "#include " in #ifndef SV68R3V6..#endif to fix Sys V R3 build on Motorola 68000 (Gerry Belanger). This one might need to be generalized to all System V prior to SVR4. ckcnet.h, 2 Jul 2001. Emilio Perea confirmed that the sco32v505x entries work on a vanilla 5.0.5 system, so I replaced sco32v505 entries (that didn't work on either mine or his) with them. makefile, 2 Jul 2001. Updated sv68r3v6 target to use xermit instead of wermit and added -DSELECT (Gerry Belanger). makefile, 3 Jul 2001. Improved error reporting for pty module. ckupty.c, 5 Jul 2001. From Jeff: . SET SSL {DSA,RSA}-CERTS-CHAIN-FILE command. . HTTP OPEN, HTTP CLOSE: persistent, separate HTTP connections. ck_ssl.c ck_ssl.h ckcdeb.h ckcftp.c ckcmai.c ckcnet.c ckcnet.h ckuath.c ckupty.c ckupty.h ckuus2.c ckuus3.c ckuus7.c ckuusr.c ckuusr.h, 5 Jul 2001. Replaced strdup() calls with makestr(). ckcnet.c, 5 Jul 2001. Added HTTP OPEN /SSL /TLS switches. ckuusr.c, 5 Jul 2001. Discovered that all the HTTP blah /ARRAY:a commands created array elements with trailing CRLFs. Fixed in about 20 repetitious code chunks in ckcnet.c, 5 Jul 2001. Looked at parsing "email-style" dates like "Fri, 26 Nov 1999 23:12:22 GMT", which are found in HTTP headers... If you remove "Fri, " and " GMT", it's parseable. It's easy enough to remove the day of week from the front, but what about the GMT? Here we run into exactly the same problems we had with FTP. We have no way of converting from GMT to local time, nor of recording the timezone if it's not GMT. Suppose we want to use an HTTP header like: Last-Modified: Mon, 06 Sep 1999 22:35:58 GMT to compare with some local date (e.g. a file date, or an /AFTER or /BEFORE date). We could do this by (a) stripping the DOW, (b) verifying that the timezone is GMT (UCS, whatever) and then stripping it, (c) parsing the result, (d) converting to internal format (sec since 19700101 00:00:00), converting the local date to internal format GMT, and (e) comparing. Similarly for asctime() format, like "Thu Jul 5 11:48:56 EDT 2001". The hard part is the timezone, not the additional date formats. Therefore, I'm deferring any work on adding /BEFORE and /AFTER switches to HTTP until at least one user asks for them. Similarly for searching HTTP headers for a specific tag and getting its value. This is easily done in a script: http open www.columbia.edu if fail stop 1 HTTP OPEN failed http /array:a get kermit/index.html /dev/null if fail stop 1 HTTP GET failed for \%i 1 \fdim(&a) 1 { .\%x := \findex(:,\&a[\%i]) if not \%x { echo BAD LINE, continue } echo TAG = "\fleft(\&a[\%i],\%x-1)" echo VAL = "\fltrim(\fsubstr(\&a[\%i],\%x+1))" } http close Or to do a tag search (e.g. for Date:), replace the loop with: .\%i = \farraylook(Date:*,&a) .\%x := \findex(:,\&a[\%i]) if not \%x { echo BAD LINE, continue } echo VAL = "\fltrim(\fsubstr(\&a[\%i],\%x+1))" (We can't use \fsplit() to break the string because the value itself might contain :'s.) It would, of course, be possible to add Yet Another Function to split a string into two pieces, breaking on the first colon (or equal sign, whatever), but we already have so many other functions like this (but slightly different), I think it would add more confusion than usefulness, especially since it would need to return two values. Discovered that HTTP arrays were (a) missing the first element, and (b) contained an empty last element. Fixed in http_mkarray() to account for the fact that the argument array is 0-based but Kermit arrays are 1-based. ckcnet.c, 5 Jul 2001. From Jeff, new FTP and HTTP variables: ftp_fd - current file descriptor ftp_security - security method negotiated ftp_cpl - command channel privacy mode ftp_dpl - data channel privacy mode http_security - security method negotiated http_fd - current file descriptor http_connected - are we? http_host - hostname of http server http_code - last status code http_message - last status message Plus change in default ciphers for FTP SRP, correction to authtype variable, which did not take into account difference between client and server mode. ckuusr.h ckuus4.c ckcftp.c ckcnet.c, 5 Jul 2001. Added missing CK_SSL #ifdefs to new ckcnet.c code. ckcnet.c, 5 Jul 2001. Some time in the past year-and-a-half I changed FOR loops to allow macros as loop variables. This turns out to work for some variable names but not others. For example, it works for i but not for x. Why? Because somewhere in the bowels of the parser, a backslash is being inserted in front of the variable name on the assumption that it's a backslash variable (which needs to be quoted). This doesn't hurt "i" because "\i" means just "\i", but "\x" is a hex-code introducer. It turns out we're not doing this in the code at all -- it's simply the consequence of the master FOR macro definition in ckuus5.c: /* FOR macro */ char *for_def[] = { "_assign _for\\v(cmdlevel) { _getargs,", "def \\\\\\%1 \\feval(\\%2),:_..top,if \\%5 \\\\\\%1 \\%3 goto _..bot,", "\\%6,:_..inc,incr \\\\\\%1 \\%4,goto _..top,:_..bot,_putargs},", "def break goto _..bot, def continue goto _..inc,", "do _for\\v(cmdlevel) \\%1 \\%2 \\%3 \\%4 { \\%5 },_assign _for\\v(cmdlevel)", ""}; which is appropriate for \%x loop variables, but not for loop variables that are macros. So we solve this by having two FOR macros, one for each case. ckuus[56].c, 5 Jul 2001. Dat Nguyen reported trouble with FTP MGET /LISTFILE: and once I figured out how to reproduce it, it only took all evening to fix it. To make a long story short: at some point the code says "src = mgetlist[mgetx];". Later, we use the src pointer for making some comparisons to decide whether a file should be skipped. However, between these two places, it was possible for mgetlist[mgetx] to be freed and reallocated, thus leaving src pointing at some random thing, which, since it didn't compare as expected, caused the file to be skipped. It was hard to find, because sometimes the old freed string pointed by src was still there (depends on your free() implementation, state of the heap, etc) so nothing seemed to be wrong. The obvious solution was to make sure the src pointer is up to date before using it. ckcftp.c, 5 Jul 2001. Added SET EXIT HANGUP { OFF, ON }, ON by default. When OFF, doclean() (invoked by doexit()) skips the section that hangs up and closes the communication device. ckcmai.c, ckuus[235x].c, 6 Jul 2001. Updated hostmode.ksc and hostmdm.ksc to fix many problems: the multiple config files feature didn't work at all, various spurious error messages came out, and it didn't do anything useful if COMMPORT was not defined (as it is not by default, so in this case I made it offer to use TAPI). 6 Jul 2001. From Jeff, 7 Jul 2001: . Minor change to FTP authtype variable: ckcftp.c, ckcnet.c. . Minor syntax adjustments: ckuusr.c, ckuus4.c. The GREP command allowed ^ and $ pattern anchors, but only because of special code in dogrep() to preprocess the pattern before sending it to ckmatch(). But if GREP can do it, why not also other GREP-like functions, such as \farraylook(), \fsearch(), \frsearch(), TYPE /MATCH:, and IF MATCH? So rather than hack up each of these commands to handle ^ and $, I added a new option to ckmatch(), opts bit 2: when set, ckmatch() allows anchors (and the absence of an anchor implies a '*'). Then I changed each of the commands and functions listed to set this bit in the ckmatch() call. Obviously the new bit must be used with caution. For example, it should NEVER be set by code that is matching filenames, or else "delete x" would delete all files whose names contained an "x", rather than just the "x" file. Anyway the changes should be safe because they affect only the commands and functions listed. ckcfns.c, ckcftp.c, ckclib.c, ckucmd.c, ckufio.c, ckuus[456x].c, 7 Jul 2001. anchored floating foo ^foo$ *foo foo$ foo* ^foo *foo* foo Split HELP WILDCARD and HELP PATTERN. ckuus2.c, 7 Jul 2001. Yesterday's changes broke \fsearch(), which works by looping through the string and calling ckmatch() each time through the loop. Since \fsearch() matches are now floating by default, then if it succeeds at all, it always succeeds on the first trip thru the loop, thus returning a match position of 1. This raises the old question about how to make ckmatch() itself return the match position -- then we wouldn't need a loop. So I went back and finally did it. \fsearch() now simply reports ckmatch()'s return value. ckclib.c, ckuus4.c, 8 Jul 2001. But now \frsearch has a problem: ^xxx searches succeed even when xxx is not at the beginning of the string. That's because we're looping backwards thru the string and calling ckmatch() repeatedly on increasingly long right substrings. But that's easy to fix: if the pattern starts with ^, there is no point searching from the right, so \frsearch(^xxx) is treated like \fsearch(^xxx). ckuus4.c, 8 Jul 2001. Then I put together a thorough test suite and uncovered some other problems, some of which had always been there. Yesterday's changes also broke commands like "sho var ftp", today's fixes fixed them. But today's code breaks at least two things; matching of: . Patterns that contain quoted metacharacters . Alternation lists that follow the second occurrence of a string, e.g. \fsearch(mno{xxx,yyy,pqr},mnozzzmnopqr) From Jeff, 9 Jul 2001... Jeff's notes: ckuusr.c ckuus4.c ckcnet.c ckuath.c typo in ckuus4.c attempts to make the http code more robust. apparently it is possible for the http server to close the socket since it did not receive a request in time but for the write() call on socket to succeed. changed http /user: to allow an empty user name. this is allowed by http authentication. although I am doing something wrong still in the http_xxx() functions in that case because the string I am generating does not match the string generated by Netscape in the case of a null username. discovered a few places in the kerberos code where an invalid kerberos config file was resulting in our passing a null pointer to the kerberos libraries. The kerberos libraries do not check pointers for validity before use. ckuus7.c ckcnet.c SHOW AUTH was not displaying IP addresses set with SET AUTH K5 ADDRESSES The Reverse DNS lookups were breaking the HTTP GET request when there was no reverse DNS entry for the ip-address. The HTTP server would timeout the connection before the DNS query would timeout. Changed the code to only perform reverse DNS lookups for HTTP OPEN if SET TCP REVERSE-DNS-LOOKUP is ON. (end Jeff's notes) Back to ckmatch()... After yesterday's changes the torture tests turn up a few problems with \fsearch(). What they all boil down to is: ckmatch() never handled backing up the pattern after failure in all cases. But this was masked by the facts that it never had to return a match position before, and that we called it in a loop, which had the same effect but (a) at far greater cost, and (b) it's the place of ckmatch() to do it, not the caller. I got some of this fixed today, but still need to do more. This is worthwhile work for many reasons, including (a) it can speed up INPUT by orders of magnitude, and (b) ckmatch will be able to return the string segment that matched the pattern, which is a powerful SNOBOL feature that will be incredibly useful in scripts. Until now, you could search for a pattern, but there was never a way to find out what the pattern matched; when this works, after a pattern-match succeeds, you can say "what did I find?" (And before I leave ckmatch(), I might want to add true regexp capability, e.g. "[0-9]+" means one or more digits.) ckclib.c, 9 Jul 2001 (to be cont'd). readblock() conflicts with a QNX6 name. Fixed by an #define in ckuus6.c. New qnx_rtp makefile target. Kirk Russell , 9 Jul 2001. Kirk also pointed out that QNX6 open() gets an error for /dev/tty with O_NONBLOCK. Added QNX6-specific code for this to do_open(), ckutio.c, 9 Jul 2001. Added QNX6 to ckuver.h. 9 Jul 2001. Sebastian Larco reported that GET /MOVE-TO:xxxx didn't work for him. For me, it dumps core. Problem: a piece of MGET list-processing code was being executed in this case, when there was no list to process. Fixed in doftpget(): ckcftp.c, 9 Jul 2001. Kirk Russell pointed out that the OpenBSD version lacked POSIX RTS/CTS and that its lockfile directory was wrong. Fixed in ckcdeb.h, ckuver.h, ckutio.c, 10 Jul 2001. Added __OpenBSD__ to SHOW FEATURES. ckuus5.c, 10 Jul 2001. Added makefile target for QNX6. 10 Jul 2001. Testing serial-port/modem transfers in OpenBSD... First at 57600bps: Upload Download Remarks Elapsed time: 06:40 06:48 Flow control lights never blinked Max window: 9 1 during upload or download. Error count: 0 0 Effective CPS: 4669 4662 Peak data rate: 4772 4723 Again at 115200: Upload Download Remarks Elapsed time: 05:24 03:34 On upload, CTS and TxD lights Max window: 13 1 blinked in lockstep; flow control Error count: 0 0 worked perfectly. On download, the Effective CPS: 5881 8872 the lights never blinked (the computer Peak data rate: 5908 9072 is faster than the modem). OpenBSD 2.5 does not let me set the speed to 230400, even though it is listed in termios.h -- the driver rejects it. Finally at 19200 bps... Upload Download Remarks Elapsed time: 20:22 20:24 On upload, flow-control lights never Max window: 5 1 blink. On download... same thing. Error count: 0 0 Again, the computer is faster than Effective CPS: 1557 1555 the modem, so we never see RTS/RxD Peak data rate: 1593 1583 action. From Jeff, 10 Jul 2001: . NT needs struct _stat rather than struct stat, ckuus7.c, ckcftp.c. . NT needs struct _utimbuf rather than struct utimbuf, ckcftp.c. . Lots of #define blah _blah" for NT C lib routines, ckcdeb.h. . HTTP should be undefined when TCPSOCKET not defined. ckcdeb.h. . Corrections to LIBS clause for linux+srp+openssl target: makefile. Jeff's notes: correction to http_get_chunked_length() which was broken by Saturday's edits to remove end of lines from http arrays I applied the changes to the SRP library to ckuath.c. I have still to implement the changes for ckcftp.c. In the process I ran into a problem with incompatible libraries that caused me to tighten the compilation rules. This resulted in a new #define be used for the NT builds: __STDC__. The side-effects of this were many. A large number of functions now need to be called by the ANSI names since the K&R names are no longer supported. See the list I added to the bottom of ckcdeb.h. The old SRP authentication code is still accessible by defining PRE_SRP_1_7_3. There is no way to set this based upon the srp header files. I am simply going to have to assume that people are using 1.7.3 and explain the changes that need to be made to the makefile entry in case they are still using an older version. (end Jeff's notes) SET EXIT HANGUP OFF still didn't work because ttclos() called tthang() (reported by Keith Doyle). Fixed in ckutio.c, 11 Jul 2001. Removed redundant -DBPS_xxx's from qnx6 makefile target, verified by Kirk Russell, 11 Jul 2001. Fix from Kirk Russell to make do_open() compile on QNX6, ckutio.c, 11 Jul 2001. FTP GET /RENAME, /MOVE-TO, and /SERVER-RENAME needed more work: . global tmpbuf[] was being overwritten, as Jeff noticed. . GET was improperly requiring /RENAME-TO arg to contain \v(filename). . Transaction log entries said rename/move failed when it succeeded. Fixed in doftpget(): ckcftp.c, 11 Jul 2001. The following now work right, including the logging: get /rename-to:justastring ckuusr.h get /move-to:../tmp3 ckuusr.c get /server-rename:delete.me ckuusr.c get /server-rename:undelete.me /rename-to:itworked delete.me get /server-rename:delete.me.again /move-to:../tmp3 undelete.me get /as-name:ASNAME /rename-to:NEWNAME ckuusr.h Changed IF MATCH back to anchored; otherwise "if match abcd bc" succeeds, which might confuse people. ckuus6.c, 11 Jul 2000. Back to ckmatch()... I fixed the problem with matches like: \fsearch(mno{xxx,yyy,pqr},abcmnozzzmnopqr) where mno matches the first mno, but zzz doesn't match any of the alternation strings, so we have to restart the search. ckclib.c, 11 Jul 2000. Now all the torture tests pass with one exception: any pattern involving '*' followed by a quoted character, for example: if match ab?yz a*\\?yz This one actually dumped core due to stack overflow. As is so often the case when code becomes ungainly and incomprehensible, the fix was not to add code, but to take code away. ckclib.c, 11 Jul 2001. Now that all the tests pass, the next thing is add more tests, and sure enough we find a new case that doesn't pass: \fsearch({ck[cuw]*.[cwh],makefile},ckutio.c) Here the alternation list is not embedded within a larger pattern -- another edge case. I'll fix it tomorrow. From Jeff: . moved Bleep constants from ckuusr.h to ckcker.h since they are not really user interface values and I didn't want to include ckuusr.h in ck_ssl.c . added code to check the version number of the OpenSSL library which is loaded with Kermit. This could be a DLL on Windows OR a shared library on Unix. OpenSSL releases are not binary compatible. Therefore, if the Kermit build does not match the library version there could be serious problems some of which might be extremely hard to track down. The default behavior is to disable SSL/TLS support but allow Kermit to continue. . Re-ordered the initialization of telnet and ssl so that the telnet options will be configured properly if there was an OpenSSL library mismatch. . Fixed end of lines for HTTP HEAD when saving the headers to a file . Moved the #define for des_fixup_parity as it is now required for new versions of OpenSSL. ckcmai.c ck_ssl.c ckcker.h ckuusr.h ckuath.c ck_crp.c ckcnet.c, , 12 Jul 2001. Yesterday's ckmatch() problem: not a ckmatch() problem. The syntax: \fsearch({ck[cuw]*.[cwh],makefile},ckutio.c) does not work because {} around a function argument is used for grouping. These work: \fsearch({{ck[cuw]*.[cwh],makefile}},ckutio.c) \fsearch("{ck[cuw]*.[cwh],makefile}","ckutio.c") (Gee if I can't remember the syntax rules, who can???) Jeff reported "show var ftp" busted again. But only in Windows: missing case in #ifdef sequence in ckmatch, in the globbing sectionw where we make "*" stop at the first directory separator. Also, ckmatch() should not have been called with the globbing flag by SHOW FUNC to begin with. SHOW MAC also called ckmatch() with globbing bit. ckclib.c, ckuus[5x].c, 13 Jul 2001. I looked into making ckmatch() record the string segment that matched the pattern, and I actually got it mostly working, but it's too costly in performance (ckmatch() can be called tens of thousands of times for a single wildcard match), and there's no good way to return the value, since all the operations that call ckmatch() already return some other kind of value (success/failure, position, etc). So let's table this until it comes up again. 13 Jul 2001. Updated ~fdc/public_html/ckermit3.html. 13 Jul 2001. From Jeff: Kerberos Keytab selection commands, ckuusr.h, ckuus[237].c, ckcnet.c, ck_ssl.c, ckuath.c, 14 Jul 2001. In doinput(), ckmatch() is called in a loop so we can get the match position. This is no longer necessary; now it can be called once with option bit 2 (floating pattern) and it returns the match position. ckuus4.c, 14 Jul 2001. ckmatch() did not do case-independent matches of [A-Z][a-z] patterns correctly. Fixed in ckclib.c, 14 Jul 2001. The new ckmatch() makes it easier to recognize additional date-time formats. Added recognition of actime() format with and without timezone (but I don't do anything about timezone conversion). cmcvtdate(): ckclib.c, 14 Jul 2001. From Jeff: For the people cannot stand the switching between terminal and command mode when APCs are used: SET TERM APC NO-INPUT, NO-INPUT-UNCHECKED In order to do this I changed the apcstatus variable to store a bitmask and not fixed values. ck_des.c ckcker.h ckuat2.h ckucns.c ckucon.c ckuus2.c ckuus3.c ckuus5.c ckuus6.c ckuus7.c ckuusr.c ckuusx.c, 15 Jul 2001. Suppose I want to handle the GMT timezone in cmcvtdate() when converting to a Kermit-format local time string. In Unix I could use localtime(), which converts GMT to local time. But localtime() takes a time_t. Given a GMT time string, then, I can parse all the fields, create a struct tm from them, and then convert to time_t somehow... Hmmm, haven't we been here before, lots of times? Perhaps once and for all we need to define a ck?fio.c API for converting GMT to local time based on Kermit-format date-time strings: char * zlocaltime(char * gmtstring) This hides all the hideous date/time APIs from the mainline code. Done in ckufio.c, ckcdeb.h, 15 Jul 2001. To add zlocaltime() for non-UNIX platforms, add a zlocaltime() routine to cku?io.c and then change the ZLOCALTIME definition in ckcdeb.h to include the new platform. Made cmcvtdate() call zlocaltime() when given an Asctime-format string that includes a GMT timezone, and therefore return a local date-time string. This won't have much effect, since we rarely encounter a Asctime+Timezone date string in real life, but you can see it work if you type, e.g.: date Sun Jul 15 15:42:47 GMT 2001 or: echo \fcvtdate(Sun Jul 15 15:42:47 GMT 2001) at the C-Kermit> prompt. This opens up some new possibilities, e.g. parsing dates in HTTP headers in an update script. It also might let us simplify some of the FTP date-time code, and it might eventually be useful in Kermit protocol too. Next, maybe I'll add more date formats, e.g. like these from e-mail: Date: Thu, 10 Nov 94 10:50:47 EST Date: Fri, 20 Oct 1995 18:35:15 -0400 (EDT) Date: Wed, 27 Mar 96 10:56:04 EST Date: Thu, 15 May 1997 00:44:25 -0400 Date: Fri, 24 Mar 2000 14:19:59 EST Date: Sun, 9 Apr 2000 06:46:46 +0100 Date: Fri, 2 Mar 2001 16:49:03 -0500 (EST) Date: Mon, 9 Apr 2001 13:05:23 EDT Date: Mon, 23 Apr 2001 13:55:29 -0600 (MDT) Date: Mon, 4 Jun 2001 09:00:44 -0400 Date: Fri, 08 Jun 2001 05:31:11 -0700 Date: Wed, 20 Jun 2001 03:00:20 GMT Date: Tue, 26 Jun 2001 13:33:21 +0200 Date: Tue, 26 Jun 2001 10:19:45 -0400 (EDT) Date: Fri, 13 Jul 2001 12:54:29 -0700 (PDT) Date: 14 Jul 2001 16:26:45 GMT Date: 14 Jul 2001 21:53:33 +0200 Date: Sat, 14 Jul 2001 11:49:29 and also maybe GMT timezone recognition for the formats we already support, and maybe a new \function() to convert GMT to local time. Then we probably could also use local-to-GMT conversions too, etc etc. Jeff added zlocaltime() to K95: ckcdeb.h (plus K95-specific modules), 16 Jul 2001. I'm getting core dumps after a series of complex pattern matches. Turns out to be a memory leak, in which I didn't check that a calculated offset could be negative. Fixed in ckmatch(): 16 Jul 2001. Changed ECHO to *not* try to preserve previous behavior with text enclosed in doublequotes. This exception to the rules was just too confusing, and would probably benefit very few people compared to the many it would confound. So in other words, now: echo "foo" prints foo without the quotes. ckuusr.c, 18 Jul 2001. Rewrote cmcvtdate() to recognize ISO 8061 format date-time strings: . Allow 'T' or 't' as date-time separator. . Allow hhmmss with no colons . Allow hhmm and hh (trailing fields omitted default to 00). . Allow Z after time == GMT/UTC . Allow GMT or UTC after time . Allow +/- hh:mm[:ss] to express timezone (GMT offset) So now we can handle (among many others): 20010718 19:21:15 20010718-19:21:15 20010718-192115 20010718_192115 20010718T192115 20010718t192115 20010718X192115 18 Jul 2001 192115 18-Jul-2001 1921 18-Jul-2001T1921 18-Jul-2001t1921 18-Jul-2001 1921Z 18-Jul-2001 1921 GMT 18-Jul-2001 1921 +0100 18-Jul-2001 1921 -0100 18-Jul-2001 1921 -0200 18-Jul-2001 1921 -03:00 18-Jul-2001 1921 -04:00:00 18-Jul-2001 1921 -0500 18-Jul-2001 1921 +2300 18-Jul-2001 1921 -2300 For now any other timezone is treated as an error (since we have no way of knowing what to do with it). It is unfortunate that we can't even recognize our own timezone but I have no idea where to start. For example ours could be EST or EDT, or it could be EST5EDT or who knows what else, plus in UNIX I don't even see any API that returns it (not all UNIXes have $TZ, and even if they do, what's the format?). This means if we get a time like "18-Jul-2001 19:21:00 EDT" we choke on EDT even if it is our own timezone. So this is a pending issue, as is the handling of formats like: Mon, 9 Apr 2001 13:05:23 EDT Tue, 26 Jun 2001 10:19:45 -0400 (EDT) (gobble and ignore day of week, discard trailing timezone comment, etc), and other fractured email-style dates. ckucmd.c, 18 Jul 2001. Added date error-string, to be queried when cmcvtdate() returns -1; made DATE command print it when cmcvtdate() gets an error. ckucmd.c, ckuusr.c, 19 Jul 2001. Added ability to recognize English days of week: . If entire date is a day name, it is replaced by the corresponding date. . If time is given, it is used; otherwise 00:00:00 is supplied. . If date starts with day name and comma, we skip past and ignore it. . Day names are Kermit keywords, so can be spelled out or abbreviated. Thus the following are accepted as valid dates: friday <-- time 00:00:00 is supplied fri <-- same as friday friday 12:34:45 <-- given time is used fri, 19 Jul 2001 <-- "fri, " stripped and ignored, 00:00:00 supplied fri, 19 Jul 2001 12:34:45 <-- ditto but given time is used friday 19 jul 2001 12:34 <-- rejected because no comma etc. Other changes required to accept RFC2822 email date-time formats: . Accept timezone "UT" == UTC == GMT (OK) . Anything trailing stuff in parens is a comment (OK) . Date-time +/- xxxx == local time +/- GMT offset (OK) . Max secs can be 60 (because of leap second) (OK) . Fractions of seconds are handled (OK) . GMT offset is always 4 digits (OK) . 2-digit year: 00-49: add 2000; 50-99: add 1900 (OK) . 3-digit year: add 1900 (OK) All this work is done but it broke a lot of previously working formats so more work is needed before I upload. 20 Jul 2001: cmcvtdate() now handles just about any conceivable ISO 8601 or RFC 822 or RFC 2822 or Asctime() date format, plus many others besides. Timezone conversion is performed: . If timezone is Z, UT, UTC, or GMT. . If GMT/UTC offset given (e.g. +0400, -0830, with some format flexibility). . For the following USA timezones as specified in RFC 2822: EDT EST CDT CST MDT MST PDT PST Plus it adds clever heuristics like, if a time is given without colons to separate the fields, e.g. "230", and if the number of digits is odd, a leading zero is implied so "230" becomes "02:30:00" rather than "23:00:00". Other niceties: . Days of week (e.g. SATURDAY) are accepted as symbolic dates. . Days of week and month names (and symbolic dates like TOMORROW) can be spelled out or abbreviated. . RFC 2822 windowing for 2-digit and 3-digit years. . AM/PM notation supported. . Fractional seconds supported, rounded to nearest second. . Day of week can be followed by date, which takes precedence, e.g. Sun, 9 Apr 2000 06:46:46 +0100 (comma optional). . Day of month is validated against calendar taking leap year into account. . Month name can now come first, as in "Jan 23, 2001". . If a string can't be parsed, the DATE command tells exactly why. Still to do: . Check each result carefully for correctness (again). . Use cmdatemsg in other contexts like \fcvtdate(), etc. . Clean up voluminous debug() calls (after more testing). . Check carefully for memory leaks. . Check what happens without ZLOCALTIME defined. . Handle non-GMT timezones in Asctime() dates. . Add an option to return the result in various formats, at least with "_" rather than " " as date-time separator. . Write a sample HTTP script that GETs stuff after a certain date. And eventually: . Adapt this code to Kermit protocol, solving the GMT problem by allowing an optional GMT offset in A-Packet dates. Not done, probably won't do: . Handling non-English day, month names. . User-defined rules for handling ambiguous cases like 10/11/2001. . The case where a fraction of a second pushes us across a date boundary (in this case I just drop the fraction). . No allowance for years after 9999. Years 0000-9999 are allowed. . In strings like "Wed, 20 Jul 2001" we don't check whether 20 Jul 2001 really is a Wednesday. Easy to do but adds overhead. . Nothing is done about medieval calendar adjustments. For docs: timezone conversions are done by underlying OS; any errors are the OS's fault. Example: