C-Kermit 9.0 Alpha Test Announcement

Frank da Cruz Columbia University fdc@columbia.edu

Current test level: 9.0.299 Alpha.02 Date: 1 February 2010 This page updated: Mon Feb 15 14:09:51 2010 EST

Work on C-Kermit has continued, on and off, since the release of C-Kermit 8.0.211 on 10 April 2004. The working version was called 8.0.212 but it will be released as C-Kermit 9.0, starting with some Alpha-test releases, then some Betas, then the final release when it's ready. The major goals of this release are (a) stability; (b) compatibility with newer OS releases and hardware; and (c) support for large files and 64-bit integers on as many platforms as possible. CLICK HERE to access and download the latest build. Some large projects that were planned have not been done (yet), including:

• IPv6. Honestly, there has been zero demand for this, and it would be a lot of work and disruption to the code base. Volunteers welcome, I guess. It could be a CS project.
• A database interface - MySQL or ODBC. For this one, there is significant demand but I haven't had a chance to even look into it yet. I hope to include it in the final 9.0 release.
• There's a looming issue with DES encryption; major vendors are removing it from their platforms, starting with Apple in Mac OS X 10.6, with Microsoft to follow suit. A secure version of Kermit can be built without DES, but in limited testing successful connections were spotty (e.g. with Kerberos 5).
• Cleaning up the Unix makefile. It has 25 years' worth of targets in it. It is very likely safe to remove most of them, since (a) most old platforms have gone away by now, or have been upgraded, due to hacking vulnerabilities; (b) the market has consolidated considerably; and (c) most of the new features of C-Kermit 9.0, such as long files, won't be of any use on older platforms and previous C-Kermit versions will remain available.
• Packages. Everybody wants an install package custom made for their own computer, Linux RPMs being the prime example but far from the only one. These will come, I suppose (especially with some Linux sites having a policy against installing any application that does not come as an RPM). In the meantime, here's a page that describes some Kermit-specific issues in package construction: ckpackages.html.

Thanks to (at least) Jeff Altman, Nelson Beebe, Gerry Belanger, Christian Corti, John Dunlap, Peter Eichhorn, Carl Friedberg, Günter Knauf, Jason Lehr, Mike Rechtman, Steven Schweda (SMS), Kinjal Shah, Andy Tanenbaum, Seth Theriault, and Martin Vorländer, and to Hewlett-Packard Company, for assistance and support.

Demonstration: HP iLO Blade Configuration

Illustrating some of C-Kermit 9.0's new features is a script in production use at Columbia University for configuring and deploying racks full of HP blade servers through their "integrated Lights Out" (iLO) management interface, bypassing the tedious and error-prone process of configuring the servers one by one through the vendor-provided point-and-click Web interface, which is ill-suited to configuring large numbers of blades. DETAILS HERE (this program is likely to change from time to time).

Platforms

These are the builds that were done on the current code base, C-Kermit 9.0.299 Alpha.01 and later. Many others were done previously (see next section) but I don't have access to all of those platforms any more.

OS and Version	Arch	Word	Target	Footprint	Features (LF = Long Files)
AIX 5.3	powerpc	32	aix	2.9MB	LF (target should work for 4.2 & later)
AIX 5.3	powerpc	32	aixg	2.6MB	LF (ditto but built with gcc)
AIX 5.3	powerpc	32	aix+ssl	4.2MB	LF, OpenSSL 0.9.7g
AIX 5.3	powerpc	32	aix+ibmssl	3.66MB	LF, IBM OpenSSL 0.9.8k (gcc)
Digital Unix 4.0F	alpha	64	osf	3.0MB	LF
FreeBSD 5.0	i386	32	freebsd	2.2MB	LF
FreeBSD 6.1	i386	32	freebsd	2.2MB	LF
FreeBSD 7.2	amd64	64	freebsd	2.4MB	LF
FreeBSD 7.2	amd64	64	freebsd+ssl	2.5MB	LF, OpenSSL 0.9.8e
HP-UX 6.5 (1989)	mc68020	32	hpux0650otcpc	2.1MB	non-ANSI C compiler, TCP/IP, curses
HP-UX 7.05	mc68020	32	hpux0700lfn	2.2MB	Long filenames, no TCP/IP
HP-UX 7.05	mc68020	32	hpux0700sftcpc	2.2MB	TCP/IP, curses, short filenames
HP-UX 8.00	mc68040	32	hpux0800o	2.0MB
HP-UX 8.00	mc68040	32	hpux0800onotcp	1.7MB	No TCP/IP
HP-UX 9.03	mc68020	32	hpux0900	2.5MB
HP-UX 9.03	mc68020	32	hpux0900m68ko	2.1MB
HP-UX 9.07	mc68020	32	hpux0900m68ko	2.5MB	(Cross compiled)
HP-UX 9.07	pa-risc	32	hpux0900o	2.4MB
HP-UX 10.20	pa-risc	32	hpux1000	2.5MB	LF
HP-UX 10.20	pa-risc	32	hpux1000gcc+ssl	3.3MB	LF, gcc. OpenSSL
HP-UX 11.11	pa-risc	32	hpux1100o	2.5MB	LF
HP-UX 11.11	pa-risc	32	hpux1100to	2.5MB	LF, Trusted HP-UX
HP-UX 11.11	pa-risc	32	hpux1100o+ssl	2.7MB	LF, OpenSSL
HP-UX 11.23	ia64	64	hpux1100o+	4.5MB	LF
Linux Centos 5	i386	32	linux	2.2MB	LF
Linux Gentoo 1.4.16	sparc64	32	linux	2.5MB	LF
Linux Gentoo 1.12.4	powerpc	32	linux	2.6MB	LF
Linux Gentoo 2.6.31	alpha	64	linux	2.9MB	(Alpha 200 4/200) LF
Linux Gentoo 2.6.31	powerpc	64	linux	2.7MB	(PowerMac G5) LF
Linux Fedora 3	i686	32	linux	2.2MB	LF
Linux Fedora 3	i686	32	linux+ssl	2.4MB	LF, OpenSSL 0.9.7a
Linux Fedora 3	i686	32	linux+krb5	2.3MB	LF, Kerberos 5
Linux Fedora 3	i686	32	linux+krb5+ssl	2.5MB	LF, Kerberos 5, OpenSSL 0.9.7a
Linux Fedora 10	i386	32	linux	2.3MB	LF
Linux Fedora 12	i386	32	linux	2.2MB	LF
Linux Fedora 12	i386	32	linux+ssl	2.4MB	LF, OpenSSL 0.9.8j
Linux Fedora 12	i386	32	linux+krb5	2.4MB	LF, Kerberos 5 ***
Linux RHEL4	i386	32	linux	2.2MB	LF
Linux RHEL4	i386	32	linux	2.2MB	LF
Linux RHEL4	i386	32	linux+krb5	2.4MB	LF, Kerberos 5 ***
Linux RHEL4	i386	32	linux+krb5+ssl	2.5MB	LF, OpenSSL 0.9.7a, Kerberos 5 ***
Linux RHEL5.4	x86_64	64	linux	2.5MB	LF
Linux RHEL5.4	x86_64	64	linux+krb5+ssl	2.4MB	LF, OpenSSL 0.9.8e, Kerberos 5 ***
Linux RHEL5.4	ia64	64	linux	4.4MB	LF
Linux RHEL5.4	ia64	64	linux+krb5+ssl	5.0MB	LF, OpenSSL 0.9.8e, Kerberos 5 ***
Slackware 12.1.0	i386	32	linux	2.2MB	LF
Linux Ubuntu 7.10	i386	32	linux	2.3MB	LF
Linux Ubuntu 9.04	i686	32	linux	2.3MB	LF
Linux Ubuntu 9.04	i686	32	linux+ssl	2.3MB	LF + OpenSSL 0.9.7l
Mac OS X 10.3.9	powerpc	32	macosx	2.5MB	LF, No UUCP*
Mac OS X 10.3.9	powerpc	32	macosx+krb5+ssl	2.8MB	LF, No UUCP; Krb5, OpenSSL 0.9.7l
Mac OS X 10.4.11	powerpc	32	macosx	2.5MB	LF, No UUCP
Mac OS X 10.4.11	powerpc	32	macosx+krb5+ssl	2.8MB	LF, No UUCP; Krb5, OpenSSL 0.9.7l
Mac OS X 10.5.8	i386**	32	macosx	2.5MB	LF, No UUCP
Mac OS X 10.5.8	i386**	32	macosx+krb5+ssl	2.8MB	LF, No UUCP; Krb5, OpenSSL 0.9.7l
Mac OS X 10.6.1	x86_64	64	macosx	2.5MB	LF, No UUCP
Mac OS X 10.6.1	x86_64	64	macosx+krb5+ssl	2.9MB	LF, No UUCP; Krb5, OpenSSL 0.9.7l
MINIX 3 1.5	i386	32	minix315	1.9MB	No UUCP; No FTP
MirBSD 10	i386	32	openbsd	2.2MB	LF
NetBSD 1.5.2	mc68030	32	netbsd	2.0MB
NetBSD 1.6	i386	32	netbsd	2.4MB	LF
NetBSD 5.0.1	i386	32	netbsd	2.1MB	LF
NetBSD 5.0.1	i386	32	netbsd+ssl	2.3MB	LF, OpenSSL 0.9.9-dev
OpenBSD 4.5	i386	32	openbsd	2.2MB	LF
SCO OSR 6.0.0	i386	32	sco_osr600	2.4MB	LF
SGI IRIX 6.5	R10000	32	irix65	2.8MB	LF
SunOS 4.1	sparc	32	sunos41gcc	2.4MB	(fixes will be in Alpha.03)
Solaris 9	sparc	32	solaris9	2.8MB	LF
Solaris 9	sparc	32	solaris9+ssl	3.0MB	LF, OpenSSL 0.9.8l
Solaris 9	sparc	64	solaris9_64	3.3MB	LF
Solaris 10	sparc	32	solaris10	2.8MB	LF
Solaris 10	i386	32	solaris10	2.6MB	LF
VMS 7.3	VAX (SIMH)	32	@ckvker n	1.7MB	No TCP/IP
VMS 7.3	VAX (SIMH)	32	@ckvker	2.2MB	UCX 5.1
VMS 7.3	VAX (SIMH)	32	@ckvker if	2.5MB	UCX 5.1 + LF†† + FTP
VMS 6.2	alpha	64	@ckvker n	2.5MB	(Alpha 3000) No TCP/IP
VMS 6.2	alpha	64	@ckvker	2.5MB	(Alpha 3000) TCP/IP UCX 4.0
VMS 7.3-2	alpha	64	@ckvker n	2.7MB	(Alpha 200 4/200) No TCP/IP
VMS 7.3-2	alpha	64	@ckvker	3.0MB	(Alpha 200 4/200) TCP/IP UCX 5.4
VMS 8.3	alpha	64	@ckvker n	2.7MB	(Alpha DS10L) No TCP/IP
VMS 8.3	alpha	64	@ckvker	3.0MB	(Alpha DS10L) TCP/IP UCX 5.6
VMS 8.3	alpha	64	@ckvker if	3.2MB	UCX 5.6 + FTP + LF
VMS 8.3	alpha	64	.. ifo "" "CK_SSL" †	3.2MB	UCX 5.6 + FTP + LF + HP SSL V1.3
VMS 8.3	alpha	64	@ckvker o	3.0MB	(AlphaServer 800) TCP/IP MultiNet 5.3
VMS 8.3	alpha	64	@ckvker ifo	3.2MB	as above + FTP + LF
VMS E8.4	alpha	64	@ckvker	2.9MB	HP TCP/IP T5.7; HP C V7.3
VMS 8.3-1H1	itanium	64	@ckvker n	4.4MB	(RX2600) No TCP/IP; HP C 7.3
VMS 8.3-1H1	itanium	64	@ckvker	4.9MB	(RX2600) TCP/IP UCX 5.6
VMS 8.3-1H1	itanium	64	@ckvker if	5.3MB	as above + FTP + LF

*	NOUUCP is included in the Mac OS X builds by popular demand; click HERE for details. To include UUCP (lockfiles and all the rest) in the Mac OS X version use "make macosx -UNOUUCP".
**	Macintosh Intel hardware is x86_64. Mac OS X 10.4 was the first to support this platform. However, 10.4 and 10.5 treat it (by default) as as a 32-bit i386. 10.6 is the first to take full advantage of its 64-bit personality.
***	Linux and probably most other Kerberos 5 builds need KFLAGS=-DNO_KRB5_INIT_ETS on the command line. I probably should just make this the default.
&dagger	If @ckvker.com ifo "" "CK_SSL" gives %CC-W-MACROREDEF complaints for des_cblock and des_key_schedule use @ckvker.com ifo "" "CK_SSL,OPENSSL_DISABLE_OLD_DES_SUPPORT".
&dagger<&dagger	The "f" option is a no-op on the VAX, where the C runtime system lacks the underlying support.

Large Files

Kermit is, first and foremost, a file-transfer program. One might expect it to be able to transfer any kind of file, but that has been decreasingly the case as file sizes began to cross the 2 gigabyte threshold.

The biggest change since C-Kermit 8.0.211 is support for large files on platforms that support them. A "large file" is one whose size is greater than 2³¹-1 (2,147,483,647) bytes (2GB-1); that is, one whose size requires more than 31 bits to represent. Before now, Kermit was able to access such files only on 100% 64-bit platforms such as Digital Unix, later known as Tru64 Unix. In the new release, Kermit takes advantage of the X/Open Single UNIX Specification Version 2 (UNIX 98) Large File Support (LFS) specification, which allows 32-bit platforms to create, access, and manage files larger than 2GB.

Accommodating large files required code changes in many modules, affecting not only file transfer, but also file management functions from directory listings to local file manipulation, plus the user interface itself to allow entry and display of large numbers. All this had to be done in a way that would not affect pure 32-bit builds on platforms that do not support large files. Here's a table showing progress so far. Entries in Yellow (32-bit builds that support 64-bit integers) and Green (64-bit builds) support large files.

OS and Version	Arch	Word	Target	Footprint	Remarks
Linux	i386	32/64	linux	2.2MB	LF OK in all PC Linuxes back to at least 1999
Linux	mips	32/64	linux	3.4MB	Large files OK
Linux	powerpc	32/64	linux	2.4MB	Large files OK
Linux	sparc	32/64	linux	2.4MB	Large files OK
Linux	alpha	64	linux	2.6MB	Large files OK in all 64-bit Linux builds
Linux	amd64	64	linux	2.4MB	Large files OK in all 64-bit Linux builds
Linux	ia64	64	linux	4.4MB	Large files OK in all 64-bit Linux builds
Linux RHEL5	x86_64	64	linux	2.4MB	Large files OK in all 64-bit Linux builds
FreeBSD 3.3	i386	32/64	freebsd	...	LF OK in all FreeBSD 3.3 and later on i386
FreeBSD 6.1	amd64	64	freebsd	2.4MB	Large files OK
FreeBSD 6.0	alpha	64	freebsd	2.8MB	Large files OK
FreeBSD 6.0	ia64	64	freebsd	...	Large files OK
MirBSD 10	i386	32/74	openbsd	2.2MB	Large files OK
NetBSD 1.5	i386	32	netbsd	...	Large files not supported by OS
NetBSD 1.6	i386	32/64	netbsd	2.4MB	Large Files OK
NetBSD 2.0	Sun 3/80	32/64	netbsd	1.9MB	mc 68030, Large files OK
NetBSD 2.x	i386	32/64	netbsd	...	Large files OK
NetBSD 3.1	i386	32/64	netbsd	2.2MB	Large files OK
NetBSD 5.0	i386	32/64	netbsd	2.1MB	Large files OK
OpenBSD 2.5	i386	32/64	openbsd	...	LF (OK in OpenBSD 2.5 and later on PC)
OpenBSD 4.5	i386	32/64	openbsd	2.2MB	Large files OK
Mac OS X 10.3.9	powerpc	32/64	macosx10	2.5MB	Large files OK
Mac OS X 10.4.11	powerpc	32/64	macosx10	2.5MB	Large files OK
Mac OS X 10.5.8	i386	32/64	macosx10	2.5MB	Large files OK
Mac OS X 10.6.1	x86_64	64	macosx10	2.5MB	Large files OK
Mac OS X 10.6.2	x86_64	64	macosx10	2.5MB	Large files OK
HP-UX 5-9	various	32	hpux0nnn...	...	No large file support
HP-UX 10.20	pa-risc	32/64	hpux1000	2.5MB	Large files OK
HP-UX 11.11i v2	pa-risc	32/64	hpux1100	2.4MB	Large files OK
HP-UX 11.23	ia64	32/64	hpux1100o		Large files OK
HP-UX 11.23i v2	ia64	32/64	hpux1100	4.3MB	Large files OK
HP Tru64 Unix 4.0F	alpha	64	tru64_40f	...	Large files OK
HP VMS / OpenVMS	VAX	32	ckvker.com	...	See note *
HP VMS / OpenVMS	alpha	64	ckvker.com	...	See note *
HP VMS / OpenVMS	ia64	64	ckvker.com	...	See note *
IBM AIX 5.1	powerpc	32/64	aix51	2.8MB	Large files supported by AIX 4.2 and later
IBM AIX 5.3	powerpc	32/64	aix53	2.8MB	Large files supported by AIX 4.2 and later
QNX 4.25	i386	32	qnx32	...	No large file support
SCO UnixWare 7.1.4	i386	32/64	uw7	3.1MB	Large files OK
SCO OSR5.0.x	i386	32	sco32v5xx		No support for large files in OS
SCO OSR6.0.0	i386	32/64	sco_osr600	2.3MB	Large files OK
SGI IRIX 6.5	MIPS	32/64	irix65, irix65g	...	Large files OK
SGI IRIX 6.4 & earlier	MIPS	32/64	irix...	...	Might work but not tested
Solaris 5-9	i386	??	solaris...	...	Untried and untested
Solaris 5-8	sparc	??	solaris...	...	Untried and untested
Solaris 9	sparc	32/64	solaris9	2.5MB	Large files OK
Solaris 9	sparc	64	solaris9_64	3.3MB	Large files OK (-xarch=generic64)
Solaris 10	sparc	32/64	solaris10	3.8MB	Large files OK
Solaris 10	i386	32/64	solaris10	2.3MB	Large files OK
Solaris 10	sparc	64	solaris10g_64	5.9MB	Large files OK (-m64)
Microsoft Windows	i386	32		3.7MB	Kermit 95, see note **

*

At the RMS level, VMS has always supported file sizes up to 1TB (512 bytes/block × 2G blocks), but the C run-time library deals in bytes, and it was restricted to 32-bit (usually signed) size values. On non-VAX architectures, VMS V7.2 added 64-bit file size/offset values to the C RTL (enabled by the _LARGEFILE macro in <decc$types.h>). C-Kermit 9.0 can be built with the "f" option to try to enable large files, but the results will probably depend on the VMS version and the C compiler and runtime version. Details to come.

**

Kermit 95 for Windows is a member of the C-Kermit family and shares most of the same code. So far Kermit 95 has not yet been adapted for large files. HOWEVER, it seems K95 2.1.3 can send and receive large files successfully, even though the numbers shown on the file-transfer display are wrong, as is the progress bar, because it does not support long integers (this was confirmed with a 4.3GB file, whose size requires more than 32 bits to express).

No information yet for anything else not mentioned above.

How to Test Large-File Transfer

There are several methods for testing large-file transfers:

• By transferring a real file that is more than 2147483648 bytes long (a file whose requires more than 31 bits to express); or to be totally sure, that is longer than 4294967296 bytes (32 bits or more). Or to be double super sure, longer than 8589934592 (33 bits).
• If you don't have such a file or there is not sufficient disk space for such a file, you can create a special kind of file that takes up one block on the disk but appears to be 4.3GB long by compiling and running THIS C PROGRAM on Linux, Solaris, HP-UX, or other Unix platform that supports large files. Kermit or FTP or any other file transfer program will transfer the result (BIGFILE) in such a way as to actually put 4.3GB (or other desired size; see source) on the wire.
• You can use Kermit's CALIBRATE feature to transfer a large file that doesn't exist. At the receiver, use RECEIVE /CALIBRATE. At the sender, use SEND /CALIBRATE:length, e.g.:

  (At remote kermit...)
  $ kermit -Y
  C-Kermit> receive /calibrate
  (Return to local kermit...)
  Ctrl-\c
  C-Kermit> send /calibrate:4300000000

  This sends a simulated file 4.3GB in length, that does not exist on the sender and will not take up any disk space on the receiver. SEND /CALIBRATE: accepts big numbers only in Kermit versions that support them (this does not include Kermit 95 on Windows).

Arithmetic with Large Integers

Because large file support requires the availability of a 64-bit signed integer data type, other aspects of C-Kermit were adapted to use it too, most notably Kermit's algebraic expression evaluator and its S-Expression handler, on all platforms that support large files (those listed as 64 or 32/64 in the Word column of the table). In fact, every Kermit command that parses a number in any field can now parse a large number.

S-Expressions can now be forced to operate with integers only, without floating-point conversion or having to explicitly truncate each result; as an example. see the revised Easter date calculation script.

Other New Features

See the C-Kermit Daily Builds page for details. Very briefly:

• Perhaps most important, modernized makefile targets for the major Unix platforms: Linux, Mac OS X, AIX, Solaris, etc. These are somewhat automated; not autoconf exactly, but they cut down significantly on redundant targets. For example, one single "linux" target works on many (hopefully all) different Linux configurations, where before different targets were required for different combinations of (e.g.) curses / ncurses / no curses; 32-bit / 64-bit; different feature sets and library locations. (Separate targets are still required for Kerberos and/or SSL builds.)
• Bigger buffers, more storage for commands, macros, scripts, strings, and filename expansion in 64-bit versions and in 32-bit versions that support long files.
• New options for the RENAME command, allowing you to rename groups of files at once, changing case of letters in the name or changing its character set, removing spaces or changing them to something else, and/or doing anchored or floating or occurrence-based string replacement, described HERE.
• Built-in FTP client for VMS. This is the same FTP client Unix C-Kermit has had since version 8.0, documented HERE, minimally adapted to VMS by SMS, supporting binary and Stream_LF file transfer only (in other words, nothing to handle RMS files), but otherwise fully functional (and scriptable) and theoretically capable of making connections secured by SSL (at least it compiles and links OK with SSL – HP SSL 1.3 in this case). In the present Alpha release, this is an optional feature requested by including the "i" option in P1 (and by including "CK_SSL" in P3 if you also want SSL). much testing is needed to determine if it should be included in the final C-Kermit 9.0 release.
• Large file support in VMS, also by SMS. VMS C-Kermit was already able to transfer large files, but the file-transfer display (numbers and progress bar) and statistics were wrong because they used ints. In the present Alpha test release, this is an optional feature requested by including the "f" option in P1.
• User-settable FTP timeout, works on both the data and control connection.
• FTP access to ports higher than 16383.
• New PUTENV command that allows Kermit to pass environment variables to subprocesses (Unix only).
• Unix C-Kermit SET TERMINAL TYPE now passes its arguments to subprocesses as an environment variable.
• New TOUCH command, many file selection options.
• New DIRECTORY command options and switches (/TOP, /COUNT; HDIRECTORY, WDIRECTORY...). To see the ten biggest files in the current directory: "dir /top:10 /sort:size /reverse *" or equivalently, "hdir /top:10 *". WDIR lists files in reverse chronological order, shorthand for "dir /sort:date /reverse".
• New command FSEEK /FIND:string-or-pattern, seeks to the first line in an FOPEN'd file that contains the given string or matching the given pattern. Example: Suppose you have a file of lines like this:

  quantity   description...

  in which the first "word" is a number and the description. Here is how to use FSEEK to quickly get the total quantity of any given item, which is passed as a parameter (either a literal string or a pattern) on the command line:

  #!/usr/local/bin/kermit +
  if not def \%1 exit 1 Usage: \fbasename(\%0) string-or-pattern
  
  .filename = /usr/local/data/items.log     # Substitute the actual filename
  set case off                              # Searches are case-independent
  fopen /read \%c \m(filename)              # Open the file
  if fail exit 1
  .total = 0                                # Initialize the total
  echo Searching "\%1"...
  
  while true {
      fseek /line /relative /find:\%1 \%c 0 # Get next line that has target
      if fail break                         # Failure indicates EOF
      fread /line \%c line                  # Read it
      if fail break                         # EOF
      increment total \fword(\m(line),1)    # Increment the total
  }
  fclose \%c                                # Close the file
  echo Total for "\%1" : \m(total)          # Print the result
  exit 0
  

  The syntax of the FSEEK command in this example indicates that each search should start relative to the current file line. Since Kermit is an interpretive language, FSEEK is a lot faster than FREAD'ing each line and checking it for the target, especially for big files. An especially handy use for FSEEK is for use with potentially huge timestamped logs, to seek directly to the date-time where you want to start processing. Some other improvements for the FOPEN/FREAD/FWRITE/FCLOSE family of commands are included also (performance, bug fixes, convenience features), listed in the change log. (Prior to 8.0.299 Alpha.02, the FSEEK /FIND: command always started from the top.)
• SET SESSION-LOG TEXT now strips out ANSI escape sequences from the session log (terminal emulation only, doesn't work with the INPUT command but maybe this can be added later).
• For interacting with POP servers over clear-text or SSL-secured connections:
  • New SSL and TLS "raw" connections (no Telnet protocol).
  • New INPUT command options for reading and capturing (perhaps while scanning) continuous incoming text, such as INPUT /NOWRAP (explained HERE).
  • New \femailaddress() command to extract the e-mail address from an Internet mail message To: or From: line, used in fetching mail from POP servers.
  • Improved date parsing commands and functions for parsing the different date formats that can appear in e-mail.
  • Production scripts for fetching mail from a secure POP server, available HERE.
• Various features added to make Kermit more useful for writing CGI scripts such as INPUT /COUNT:n to INPUT exactly n characters (useful for reading form data).
• New \fpictureinfo() function for getting orientation and dimensions of JPG and GIF images, described HERE.
• New \fgetpidinfo() function for testing whether a given process exists.
• \fkwdvalue() function fixed to allow multiword values.
• New function \fcount(s1,s2) to tell the number of occurrences of s1 in s2.
• New \flopx() function returns rightmost field from string (such as a file's extension).
• New function \ffunction(s1) to tell whether a built-in s1 function exists.
• New \fsqueeze(s1) function removes leading and trailing whitespace from string s1, changes tabs to spaces, squeezing all runs of repeated whitespace characters to a single space (Alpha.02).
• Compact substring notation: \s(somestring[12:18]) is the same as \fsubstring(\m(somestring),12,18), i.e. the substring starting at position 12, 18 characters long. \s(somestring[12_18]) means characters 12 through 18 of the string (7 characters).
• The string indexing functions now accept an optional trailing argument specifying the occurrence number of the target string. Likewise, \fword(0) can fetch words from the right as well as the left.
• The COPY command in Unix C-Kermit has a new /PRESERVE switch, equivalent to Unix "cp -p".
• ASKQ /ECHO:c can be used to make the characters the user types echo as character c, e.g. asterisk when typing a password.
• IF LINK filename to test if the filename is a symlink.
• Ctrl-K, when typed at the command parser, replaces itself with most recently entered file specification.
• In Unix, the ability to log a terminal session to a serial port, for use with speaking devices or serial printers; described HERE. Also for the same purpose, SET SESSION-LOG NULL-PADDED-LINES for a speech synthesizer than needed this.
• Adaptation to OpenSSL 0.9.8 and, hopefully, 1.0.0.
• Improved pattern matching in many commands for both strings and filenames.
• Various minor new features, plus numerous bug fixes and speedups.

Incompatibilities

A top priority for new Kermit software releases has always been backwards compatibility. A script written for a previous Kermit release should run the same way in the new release.

There's one exception this time. The \fsplit() function is incredibly handy, it can do almost anything, up to and including parsing a LISP program (the underlying code is the basis of the S-Expression interpreter). But did you ever try to use it to parse (say) a Tab-Separated-List (TSV file) or Comma-Separated-List (CSV)? It works as expected as long as the data contains only 7-bit characters. But if your data contains (say) Spanish or German or Russian text written in an 8-bit character set such as ISO 8859-1, every 8-bit character (any value 128-255) is treated as a break character. This is fixed in C-Kermit 9.0 by treating all 8-bit bytes as "include" characters rather than break characters, a total reversal of past behavior. I don't think it will affect anyone though, because if this had happened to anyone, I would have heard about it!

Since most standard 8-bit character sets have control characters in positions 128-160, it might have made more sense to keep 128-160 in the break set, but with the proliferation of Microsoft Windows code pages, there is no telling which 8-bit character is likely to be some kind of text, e.g. “smart quotes”.

And a Loose End: Using External File-Transfer on Secure Connections

After C-Kermit 8.0.212 Dev.27 (2006/12/22), I spent a big chunk of time trying to solve a particular problem that some of you have complained about and others might be familiar with: If you use C-Kermit to make a secure Telnet connection to another host (e.g. with Telnet SSL/TLS, Kerberos, or SRP) and then attempt to transfer a file using an external protocol such as Zmodem, it doesn't work.

That's because as currently coded, C-Kermit simply starts the external protocol in a fork with its standard i/o redirected to the connection. This completely bypasses the encryption and decryption that is done by C-Kermit itself, and of course it doesn't work. The routine that handles this is ttruncmd() in ckutio.c.

In order to allow (say) Zmodem transfers on secure connections, it is necessary for C-Kermit to interpose itself between the external Zmodem program and the connection, decrypting the incoming stream before feeding it to Zmodem and encrypting Zmodem's output before sending out the connection.

In principal, this is simple enough. We open a pseudoterminal pair ("master" and "slave") for Zmodem's i/o and we create a fork and start Zmodem in it; we read from the fork pty's standard output, encrypt, and send to the net; we read from the net, decrypt, and write to the fork pty's standard input.

In practice, it's not so simple. First of all, pseudoterminals (ptys) don't seem to interface correctly with certain crucial APIs, at least not in the OS's I have tried (Mac OS X, Linux, NetBSD, etc), such as select(). And i/o with the pty often – perhaps always – fails to indicate errors when they occur; for example, when the fork has exited.

But, even after coding around the apparent uselessness of select() for multiplexing pty and net, and using various tricks to detect when the external protocol exits and what its exit status is, I'm still left with a show-stopping problem: I just simply can not download (receive) a file with Zmodem, which is the main thing that people would probably want to do. I can send files just fine, but not receive. The incoming stream is delivered to Zmodem (to the pty slave) but upon arrival at the Zmodem process itself, pieces are always missing and/or corrupt. Yet I can receive files just fine if I use Kermit itself (C-Kermit or G-Kermit) as the external protocol, rather than Zmodem.

I can think of two reasons why this might be the case:

1. Zmodem sends all 8-bit bytes and control codes in the clear, and maybe the pty is choking on them because it thinks it is a real terminal.

But Zmodem puts its controlling terminal into raw mode. And C-Kermit puts the pty into raw mode too, just for good measure. If any 0xFF codes are in the Zmodem data stream, and it's a Telnet session, Kermit does any needed byte stuffing/unstuffing automatically. Anyway, if I tell Zmodem to prefix everything, it makes no difference.

2. Zmodem is a streaming protocol and perhaps the pty driver can't keep up with a sustained stream of input at network speeds. What would be the method of flow control?

I can vary the size of the i/o buffers used for writing to the pty, and get different effects, but I am not able to get a clean download, no matter what buffer size I use. write()'ing to the pty does not return an error, and I can't see the errors because they happen on the master side. It's as if the path between the pty slave and master lacks flow control; I deliver a valid data stream to the pty slave and the master gets bits and pieces. This impression is bolstered somewhat by the "man 7 pty" page in HP-UX, which talks about some special modes for ptys that turn off all termio processing and guarantee a flow-controlled reliable stream of bytes in both directions – a feature that seems to be specific to HP-UX, and exactly the one we need everywhere.

Well, in Pass One I used C-Kermit's existing pty routines from ckupty.[ch], which are well-proven in terms of portability and of actually working. They are currently used by SET HOST /PTY for making terminal connections to external processes. But these routines are written on the assumption that the pty is to be accessed interactively, and maybe they are setting the fork/pty arrangement up in such a way that that's not suitable for file transfer. The Pass One routine is called xttptycmd() in ckutio.c.

So in Pass Two I made a second copy of the routine, yttptycmd(), that manages the pty and fork itself, so all the code is in one place and it's simple and understandable. But it still doesn't work for Zmodem downloads. In this routine, I use openpty() to get the pty pair, which is not portable, so I can have access to both the master and slave pty file descriptors. This version can be used only a platforms that have openpty(): Linux, Mac OS X, NetBSD, etc.

In Pass Three, zttptycmd(), I tried using pipes instead of ptys, in case ptys are simply not up to this task (but that can't be true because if I make a Telnet or SSH connection into a host, I can send files to it with Zmodem, and the remote Zmodem receiver is, indeed, running on a pty). But pipes didn't work either.

In Pass Four, I extracted the relevant routines into a standalone program based on yttptycmd() (the openpty() version, for simplicity), which I tested on Mac OS X, the idea being to rule out any "environmental" effects of running inside the C-Kermit process. There was no difference -- Kermit transfers (with C-Kermit itself as the external protocol) worked; Zmodem transfers (neither sz or lsz) did not.

Well, it's a much longer story. As the external protocol, I've tried rzsz, crzsz, and lrzsz. We know that some of these have quirks regarding standard i/o, etc, which is one of the reasons for using ptys in the first place, and i/o does work – just not reliably. Anyway, the 1100 lines or so of ckc299.txt, starting just below where it says "--- Dev.27 ---" tell the full story. At this point I have to give up and move on; it might be more productive to let somebody else who has more experience with ptys take a look at it – if indeed anyone still cares about being able to do Zmodem transfers over secure Telnet connections.

C-Kermit 9.0 contains the three new routines (and some auxiliary ones), but they are not compiled or called unless you build it specially:

make targetname KFLAGS=-DXTTPTYCMD (builds with xttptycmd())
make targetname KFLAGS=-DYTTPTYCMD (builds with yttptycmd())
make targetname KFLAGS=-DZTTPTYCMD (builds with zttptycmd())

These are all in ckutio.c. As noted, the second one works only for Linux, FreeBSD, NetBSD, and Mac OS X, because it uses non-POSIX, non-portable openpty(). If you want to try it on some other platform that has openpty(), you can build it like this:

make targetname "KFLAGS=-DYTTPTYCMD -DHAVE_OPENPTY"

(and let me know, so I can have HAVE_OPENPTY predefined for that platform too). The best strategy to get this working, I think, would be to concentrate on yttptycmd(), which is the simpler of the two pty-based routines. If it can be made to work, then we'll see if we can retrofit it to use the ckupty.c routines so it will be portable to non-BSD platforms.

By the way, if you build with any of [XYZ]TTPTYCMD defined, then the selected routine will always be used in place of ttruncmd(). This is to allow testing on all kinds of connections, not just secure ones, in both local and remote mode. Once the thing works, if it ever does, I'll add the appropriate tests and/or commands.

By default, in the initial test release, C-Kermit 9.0 uses ttruncmd() on serial connections and ttyptycmd() on network connections. Even when a network connection is not encrypted, Kermit still needs to handle the network protocol, e.g. the quoting of 0xff bytes on Telnet connections.

– Frank da Cruz   fdc@columbia.edu

C-Kermit 9.0 / The Kermit Project / Columbia University / kermit@columbia.edu