C-Kermit 9.0 Alpha Test
Platform-independent, transport-independent, scriptable
communication software
Frank da Cruz
Columbia University
fdc@columbia.edu
|
|
Current test level: 9.0.299 Alpha.09
Date: 21 September 2010
This page updated:
Thu Mar 10 15:41:27 2011
(Eastern USA time)
|
Work on C-Kermit has continued, on and off, since
the release of C-Kermit 8.0.211 on 10 April 2004. After that, the working
version was called 8.0.212, but it will be released as C-Kermit 9.0,
starting with some Alpha-test releases, then some Betas, then the final
release when it's ready. The major goals of this release are
(a) compatibility with newer OS releases and hardware;
(b) support for large files and 64-bit integers on as many
platforms as possible; (c) stability; and
(d) a great many VMS-specific improvements. Other new
features are listed below. Perhaps the biggest change
of all is a...
NEW LICENSE
Beginning in Alpha.10 C-Kermit will
have the Open Source Simplified BSD
License, as will subsequent Alpha and Beta tests and the final
C-Kermit 9.0 release.
CLICK HERE to
access and download the latest build.
Thanks to (at least) Jeff Altman, Ian Beckwith, Nelson Beebe, Gerry
Belanger, Joop Boonen, Rob Brown, Christian Corti, John Dunlap, Peter
Eichhorn, Carl Friedberg, Günter Knauf, Jason Lehr, Lewis McCarthy, Gary
Mills, Mike Rechtman, Steven Schweda (SMS), Kinjal Shah, Andy Tanenbaum,
Seth Theriault, Martin Vorländer, and Eric Weaver for assistance, and to
Hewlett-Packard Company for support.
What's New in Alpha.09
- New December 8, 2010: ksitemap, a C-Kermit 9.0 script to build
sitemap.xml for a website, complete with Google image extensions
(this is the file used by webmasters to get their sites crawled and indexed
optimally).
- Builds for Solaris 9 and later now use streams ptys rather then the old
BSD-style ptys. Thanks to Gary Mills for this one, who noticed that he
couldn't have more than 48 C-Kermit SSH sessions going at once and figured out
why.
- As noted below DES encryption is being retired
from many platforms and libraries that once used it. I changed the Solaris
and Linux OpenSSL builds to account for this by testing for it. I probably
should also add a OMITDES option to omit DES even if it is installed.
- I changed the Linux build to test for the OpenSSL version (like the
Solaris version already did), rather than assuming OpenSSL 0.9.7.
- A couple minor changes for Tru64 Unix 5.1B from Steven Schweda but we
still have some trouble on that platform. As a workaround "make osf1" can
be used there.
- Unix makefile and man page are now included in the Zip distribution.
Meanwhile I've been getting complaints that C-Kermit's "ssh" command doesn't
work in various Linux distributions. Turns out that some of them,
reportedly including Ubuntu and Centos, have disabled C-Kermit's
pseudoterminal support. Solution: get C-Kermit from here and build it
yourself with "make linux" (or "make linux+ssl", etc).
.
What's New in Alpha.08
- \fjoin(), which is the inverse function of fsplit()
now accepts CSV and TSV as a second argument, to transform an array into
a comma-separated or tab-separated value list, as described
HERE.
- Even in 2010, Unix distributions continue to change their UUCP lockfile
conventions. Alpha.08 contains support from Joop Boonen for OpenSuse
≥ 11.3 and recent Debian, which no longer have baudboy.h, which
first appeared in Red Hat 7.2 in 2003.
What's New in Alpha.07
This is a minor test release, mostly bug fixes:
- From Lewis McCarthy:
Based on code inspection, C-Kermit appears to have an SSL-related security
vulnerability analogous to that identified as CVE-2009-3767 (see e.g.
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3767).
I'm attaching a patch for this issue relative to the revision of ck_ssl.c
obtained from a copy of http://www.columbia.edu/kermit/ftp/test/tar/x.zip
downloaded on 2010/07/30, which I believe is the latest.
When this flaw was first widely publicized at last year's Black Hat
conference, it was claimed that some public certificate authorities had
indeed issued certificates that could be used to exploit this class of
vulnerability. As far as I know they have not revealed specifically which
public CA(s) had been found issuing such certificates.
Some references:
- Peter Eichhorn reported that "RENAME ../x ." didn't work; fixed
now.
- If only one file is FOPEN'd, FCLOSE given with no arguments would close
it; this was a "convenience feature" that turned out to be dangerous.
For safety FCLOSE has to require a specific channel number or the word ALL.
- Added \fstrcmp(s1,s2,case,start,length), which has the
advantage over IF EQU,LGT,LLT that case senstivity can be specified as a
function arg, and also substrings can be specified.
- Fixed a subtle flaw in the CSV feature that was
added in Alpha.06, namely that if the last item in a comma separated list
was enclosed within doublequotes with a trailing space after the closing
doublequote, a spurious empty final element would be created in the result
array.
What's New in Alpha.06
New ability to read and parse
Comma-Separated-Value (CSV) files,
described
HERE.
What's New in Alpha.05
- New built-in functions:
- \fcvtcsets(string,cs1,cs2)
- Function to convert a string from one character set to another.
- \fdecodehex(string[,prefix])
- Function to decode a string containing hex escapes.
- \fstringtype(string)
- Function to tell whether a string is 7-bit, 8-bit, or UTF-8.
For the motivation for these features and an application that uses them to
analyze web logs, see the Weblog script below.
- MIME Character-Set Names: A new equivalence between MIME names
and Kermit names for character sets, with a new table showing the supported
sets HERE (this feature is also
illustrated in the Weblog script).
- Lazy IF Conditions: Third, now you can do this:
- define foo some number
if foo command
- instead of this:
- define foo some number
if \m(foo) command
Of course the old way still works too. But watch out because if the
variable name is the same as a symbolic IF condition (for example COUNT),
it won't do what you expected.
New Demonstrations for Alpha.05 - Webmaster Tools
-
The Weblog Script
- Reads a web log, extracts the Google searches, normalizes the search
strings, and prints the top 20 searches, along with their counts.
Documented HERE.
-
The Amazon Script
- Reads an Amazon Associate orders report and lists the products according
to the number of orders for each, or the number of clicks on each.
Both of the scripts illustrate (among other things) how to read
Tab-Separated-Value (TSV) files.
What's New in Alpha.03 and Alpha.04
General improvements and platform-independent fixes:
- Escape sequences are now stripped from text-mode session logs not only
in CONNECT sessions but also in whatever is logged by the
INPUT command; described in the next
section.
- New commands for selectively issuing progress or debugging messages from
scripts, also described in the next section.
- Fix from John Dunlap to prevent the fixed
packet-timeout interval from going to an unexpected value.
For UNIX:
- Alpha.04 fixes a problem with FTP connections made from 64-bit Unix
platforms. All the other changes in this section were to Alpha.03.
- Relaunching a closed SSH connection with the CONNECT command is now
possible, as it always has been with Telnet and other connection types;
suggested by Peter Eichhorn (needs testing).
- A symbol conflict fixed that prevented successful build on
FreeBSD 8.0.
- Fixes from Christian Corti for building on SunOS 4.1.
- New aixg target for building on AIX with gcc.
- New aix+ibmssl target. This is nice because the
IBM-supplied SSL libraries and header files are in a known location; no need
to set environment variables
giving their locations.
For VMS:
- "Large File Support" is now included by default on Alpha and IA64 hardware
on VMS 7.3 and later, and it should work much better than before.
- Kermit's internal FTP client is now included by default in any build
that also includes TCP/IP networking. At present, the FTP client seems to
work well for binary-mode transfers; text (ASCII) mode transfers still need
some work. In builds that also include Secure Sockets Layer (SSL) security
(next item) the FTP client should be able to make securely authenticated and
encrypted connections.
- In network builds that request OpenSSL support, e.g.:
$ @ckvker "" "" "CK_SSL"
the OpenSSL version is detected automatically and the appropriate
compile-time options are emitted (such as
OPENSSL_DISABLE_OLD_DES_SUPPORT).
- Preliminary / limited support for the ODS-5 file system on VMS 7.2 and
later, Alpha and Itanium only (needs testing): Filenames can be mixed case
and can be longer.
- Support for older and older VMS versions.
In the VMS build procedure,
CKVKER.COM, the "i" option in P1
now means
don't include the internal FTP client, and the "f" option
means
do not include "Large File" support. Large File support in VMS
really only applies to the file-transfer display and statistics, which would
go out of whack as soon as the byte count overflowed 31 bits because this is
C-Kermit, built with the C compiler and the C library (runtime system),
which did not support long integers until VMS 7.3.
Special thanks to Steven M Schweda,
Martin Vorländer, and Rob Brown for most of the VMS improvements in
Alpha.03.
Demonstration: HP Switch Configuration Backup
A common use for Kermit software is to make automated backups of the
configuration of network switches and routers, such as those made by Cisco
or Hewlett-Packard (although
tftp
can be used for this, it is not available in all such devices; Kermit,
however, works with those that have tftp as well as those that don't).
Typically a backup can be done by making a Telnet, SSH, or serial connection
to the device with Kermit and giving a command such as "show config" at the
command-line prompt of the device with Kermit's session log activated. The
result is a list of the commands that were used to establish the current
configuration, suitable for feeding back to the device's console (e.g. with
C-Kermit's TRANSMIT command) to reestablish the same configuration or to
duplicate it on another device.
At an HP installation it was noted, however, that while the HP switches
(various ProCurve models) produced the desired list of commands, they were
interspersed with escape sequences for special effects, thus rendering the
recorded sessions unsuitable for feeding back into the switches.
C-Kermit 9.0 introduces a new feature to strip the offending sequences out
of a session log, leaving just the text. The command SET SESSION-LOG
TEXT activates this feature. In C-Kermit 9.0 Alpha.02 and earlier,
escape sequence stripping occurred only while logging interactive (CONNECT)
sessions; beginning with Alpha.03 it is done also for data that is read by
INPUT commands and therefore works for scripts too.
A sample HP Switch Configuration Backup script is HERE,
and its data file is HERE. This script also illustrates some other new features of Alpha.03:
MESSAGE text - This lets you put debugging messages in
your script that can be displayed or not, according to SET DEBUG MESSAGE
(next item). This way you don't have to change your script for debugging.
Hint: In Unix, invoke the script like this:
$ DEBUG=1 scriptname arg1 arg2...
and then include the following command in your script:
if defined \$(DEBUG) set debug message on
- SET DEBUG MESSAGE {
ON, OFF, STDERR }
- ON means MESSAGE commands should print to standard output; OFF means
they shouldn't print anything; STDERR means the messages should be printed
to stderr.
DEBUG MESSAGE is OFF by default, i.e. unless you SET it to ON or STDERR.
- IF DEBUG command
- Executes the command if SET DEBUG MESSAGE is not OFF.
- The \v(lastcommand) variable
- This variable contains the previous command. You can use it in
debugging and error message to show (for example) exactly what the command
was that just failed, without having to make a copy of the command:
set host somehost.somecompany.com
if fail exit 1 "FATAL - \v(lastcommand)"
which, if the SET HOST command fails, prints "FATAL - set host
somehost.somecompany.com" and then exits with status 1 (which normally
indicates failure).
Demonstration: HP iLO Blade Configuration
THIS
DOCUMENT describes a script in production use at Columbia University
for configuring and deploying racks full of HP blade servers through their
"integrated Lights Out" (iLO) management interface, bypassing the tedious
and error-prone process of configuring the servers one by one through the
vendor-provided point-and-click Web interface, which is ill-suited to
configuring large numbers of blades. The script illustrates some of
C-Kermit 9.0's new features; source code is available through the link. The
code is apt to change from time to time as new requirements surface.
Demonstration: IBM/Rolm/Siemens CBX Management
THIS
DOCUMENT describes a suite of scripts (some in production, some in
development) used to manage the Columbia campus 20,000-line main telephone
switch, along with about 10 satellite switches at off-campus locations.
These switches are 1980s technology*, their management consoles are serial
ports. Access is via Telnet to reverse terminal servers. The scripts
allow for interactive sessions as well as automatic production (and in some
cases formatting) of different reports required by different groups at
different intervals. These scripts replace a whole assortment of ad-hoc
ProComm ASPECT scripts that were scattered all over the place, with
passwords embedded. The new scripts are intended to be run from a
centralized server where there is a single well-secured configuration file,
and where they can be used on demand, or in cron jobs. They are modular so
code duplication is minimal. The source code for these scripts is not
presently public but if anybody is interested, something can be arranged
(contact me at
fdc@columbia.edu).
__________________________
*
| Of course the University is deploying new technology but the but the old
system will be used in parallel for some time to come.
|
So far C-Kermit 9.0 has been built and tested on Linux, Mac OS X, FreeBSD,
NetBSD, OpenBSD, MirBSD, HP-UX, AIX, MINIX, SCO OpenServer, SGI IRIX, SunOS
and Solaris, OSF/1 (Tru64 Unix), and DEC/Compaq/HP (Open)VMS, on a variety
of 32-bit and 64-bit hardware.
The builds done so far on the current code base, C-Kermit 9.0.299 Alpha.01
and later, are shown in
Table 1. See
the table for details of each build — OS and version, architecture,
word size, build procedure, resulting binary size, and any notes.
Large Files
Kermit is, first and foremost, a file-transfer program. One might expect it
to be able to transfer any kind of file, but that has been decreasingly the
case as file sizes began to cross the 2 gigabyte threshold.
The biggest change since C-Kermit 8.0.211 is support for large files on
platforms that support them. A "large file" is one whose size is greater
than 231-1 (2,147,483,647) bytes (2GB-1); that is,
one whose size requires more than 31 bits to represent. Before now, Kermit
was able to access such files only on 100% 64-bit platforms such as Digital
Unix, later known as Tru64 Unix. In the new release, Kermit takes advantage
of the X/Open Single UNIX Specification Version 2 (UNIX 98) Large File
Support (LFS) specification, which allows 32-bit platforms to create,
access, and manage files larger than 2GB.
Accommodating large files required code changes in many modules, affecting
not only file transfer, but also file management functions from directory
listings to local file manipulation, plus the user interface itself to allow
entry and display of large numbers. All this had to be done in a way that
would not affect pure 32-bit builds on platforms that do not support large
files. Large file support is summarized in the Table of
Platforms;
entries in Yellow (32-bit builds that support 64-bit integers) and Green
(64-bit builds) support large files.
Note that VMS C-Kermit and Kermit 95 for Windows have always
been able to transfer large files. However their user interface used 32-bit
integers for statistics and the file transfer display. In C-Kermit 9.0
Alpha.03, VMS C-Kermit on 64-bit platforms (Alpha and Itanium) should now
give correct statistics and progress displays. (We'll see about Kermit 95
later.)
How to Test Large-File Transfer
Several methods are available for testing large-file transfers:
- By transferring a real file that is more than 2147483648 bytes long (a
file whose length requires more than 31 bits to express); or to be totally
sure, that is longer than 4294967296 bytes (32 bits or more). Or to be
double super sure, longer than 8589934592 (33 bits).
- If you don't have such a file or there is not sufficient disk space for
such a file, you can create a special kind of file that takes up one block
on the disk but appears to be 4.3GB long by compiling and running THIS
C PROGRAM on Linux, Solaris, HP-UX, or other Unix platform that
supports large files. Kermit or FTP or any other file transfer program will
transfer the result (BIGFILE) in such a way as to actually put
4.3GB (or other desired size; see source) on the wire.
-
You can use Kermit's CALIBRATE feature to transfer a large file that
doesn't exist. At the receiver, use RECEIVE /CALIBRATE. At the
sender, use SEND /CALIBRATE:length, e.g.:
(At remote kermit...)
$ kermit -Y
C-Kermit> receive /calibrate
(Return to local kermit...)
Ctrl-\c
C-Kermit> send /calibrate:4300000000
This sends a simulated file 4.3GB in length, that does not exist on the
sender and will not take up any disk space on the receiver. SEND
/CALIBRATE: accepts big numbers only in Kermit versions that support
them (this does not include Kermit 95 on Windows). This method tests
only Kermit's ability to express and understand large file sizes, but does
not test Kermit's file-system interface, since no files are involved.
Arithmetic with Large Integers
Because large file support requires the availability of a 64-bit signed
integer data type, other aspects of C-Kermit were adapted to use it too,
most notably Kermit's algebraic expression evaluator and its
S-Expression interpreter, on all platforms that
support large files (those listed as 64 or 32/64 in the
Word column
of the
table). In fact, every Kermit
command that parses a number in any field can now parse a large number.
S-Expressions can now be forced to operate with integers only, without
floating-point conversion or having to explicitly truncate each result;
as an example. see the revised Easter date
calculation script.
Other New Features
See the
C-Kermit Daily Builds
page for details. Very briefly:
- Perhaps most important, modernized makefile targets for the major
Unix platforms: Linux, Mac OS X, AIX, Solaris, etc. These are somewhat
automated; not autoconf exactly, but they cut down significantly on
redundant targets. For example, one single "linux" target works on many
(hopefully all) different Linux configurations, where before different
targets were required for different combinations of (e.g.) curses / ncurses
/ no curses; 32-bit / 64-bit; different feature sets and library locations.
(Separate targets are still required for Kerberos and/or SSL builds, but
they are "subroutinized".)
- Bigger buffers, more storage for commands, macros, scripts,
strings, and filename expansion in 64-bit versions and in 32-bit versions that
support large files.
- New options for the RENAME command, allowing you to rename groups
of files at once, changing case of letters in the name or changing its
character set, removing spaces or changing them to something else, and/or doing
anchored or floating or occurrence-based string replacement, described HERE.
- Built-in FTP client for VMS. This is the same FTP client Unix C-Kermit has had since version
8.0, minimally adapted to VMS by SMS, supporting binary and Stream_LF
file transfer only (in other words, nothing to handle RMS files), but
otherwise fully functional (and scriptable) and
theoretically capable of making connections secured by SSL (at least
it compiles and links OK with SSL – HP SSL 1.3 in this case). In the
present Alpha release, this is an optional feature requested by including
the
i
option in P1 (and by including CK_SSL
in
P3 if you also want SSL, and then also
OPENSSL_DISABLE_OLD_DES_SUPPORT
if necessary). Much
testing is needed to determine if it should be included in the final
C-Kermit 9.0 release.
- Large file support in VMS, also by SMS. Alpha and Itanium only
(not VAX). VMS C-Kermit was already able to transfer large files, but the
file-transfer display (numbers and progress bar) and statistics were wrong
because they used ints. In the present Alpha test release, this is an
optional feature requested by including the
f
option in P1.
- User-settable FTP timeout, works on both the data and control
connection.
- FTP access to ports higher than 16383.
- New PUTENV command that allows Kermit to pass environment variables
to subprocesses (Unix only).
- Unix C-Kermit SET TERMINAL TYPE now passes its arguments to subprocesses
as an environment variable.
- New TOUCH command, many file selection options.
- New DIRECTORY command options and switches (/TOP, /COUNT; HDIRECTORY,
WDIRECTORY...). To see the ten biggest files in the current directory:
dir /top:10 /sort:size /reverse *
or equivalently, "hdir /top:10 *".
WDIR lists files in reverse chronological order, shorthand for
"dir /sort:date /reverse".
- New command FSEEK /FIND:string-or-pattern, seeks
to the first line in an FOPEN'd file that contains the given string or
matching the given pattern. Example: Suppose you have a file of lines like
this:
quantity description...
in which the first "word" is a number, and a description (for example, the
name of an item). Here is how to use FSEEK to quickly get the total
quantity of any given item, which is passed as a parameter (either a literal
string or a pattern) on the command line:
#!/usr/local/bin/kermit +
if not def \%1 exit 1 Usage: \fbasename(\%0) string-or-pattern
.filename = /usr/local/data/items.log # Substitute the actual filename
set case off # Searches are case-independent
fopen /read \%c \m(filename) # Open the file
if fail exit 1 "\m(filename): \v(errstring)" # Fail: exit with error message
.total = 0 # OK: Initialize the total
echo Searching "\%1"...
while true {
fseek /line /relative /find:\%1 \%c 0 # Get next line that has target
if fail break # Failure indicates EOF
fread /line \%c line # Read it
if fail break # (shouldn't happen)
increment total \fword(\m(line),1) # Increment the total
}
fclose \%c # Close the file
echo Total for "\%1" : \m(total) # Print the result
exit 0
The syntax of the FSEEK command in this example indicates that each search
should start relative to the current file line. Since Kermit is an
interpretive language, FSEEK is a lot faster than FREAD'ing each line and
checking it for the target, especially for big files. An especially handy
use for FSEEK is for use with potentially huge sequentially timestamped
logs, to seek directly to the date-time where you want to start processing.
Some other improvements for the FOPEN/FREAD/FWRITE/FCLOSE family of commands
are included also (perfomance, bug fixes, convenience features), listed in
the change log. (Prior to 9.0.299 Alpha.02,
the FSEEK /FIND: command always started from the
top.)
- SET SESSION-LOG TEXT now strips out ANSI escape sequences from the
session log.
- For interacting with POP servers over clear-text or SSL-secured
connections:
- New SSL and TLS "raw" connections (no Telnet protocol).
- New INPUT command options for reading and capturing (perhaps while
scanning) continuous incoming text, such as INPUT /NOWRAP (explained HERE).
- New \femailaddress() command to extract the e-mail address
from an Internet mail message To: or From: line, used in fetching mail from
POP servers.
- Improved date parsing commands and functions for parsing the different
date formats that can appear in e-mail.
- Production scripts for fetching mail from a secure POP server,
available HERE.
- Various features added to make Kermit more useful for writing CGI
scripts such as INPUT /COUNT:n to INPUT exactly
n characters (useful for reading form data).
- New \fpictureinfo() function for getting orientation and
dimensions of JPG and GIF images, described HERE.
- New \fgetpidinfo() function for testing whether a given process
exists.
- \fkwdvalue() function fixed to allow multiword values.
- New function \fcount(s1,s2) to tell the number of occurrences
of s1 in s2.
- New \flopx() function returns rightmost field from string (such
as a file's extension).
- New function \ffunction(s1) to tell whether a built-in
s1 function exists.
- New \fsqueeze(s1) function removes leading and trailing
whitespace from string s1, changes tabs to spaces, squeezing each
run of repeated whitespace characters to a single space (Alpha.02).
- Compact substring notation: \s(somestring[12:18]) is the same
as \fsubstring(\m(somestring),12,18), i.e. the substring starting
at position 12, 18 charcters long. \s(somestring[12_18]) means
characters 12 through 18 of the string (7 characters).
- The string indexing functions now accept an optional trailing argument
specifying the occurrence number of the target string. Likewise,
\fword() can fetch words from the right as well as the left.
- The COPY command in Unix C-Kermit has a new /PRESERVE switch, equivalent
to Unix "cp -p".
- ASKQ /ECHO:c can be used to make the characters
the user types echo as the character c, e.g. asterisk when typing a
password.
- IF LINK filename to test if the filename is a symlink.
- Ctrl-K, when typed at the command parser, replaces itself with most
recently entered file specification.
- In Unix, the ability to log a terminal session to a serial port,
for use with speaking devices or serial printers; described
HERE.
Also for the same purpose, SET SESSION-LOG NULL-PADDED-LINES for a speech
synthesizer than needed this.
- Adaptation to OpenSSL 0.9.8 and, hopefully, 1.0.0.
- Improved pattern matching in many commands for both strings and filenames.
- Various minor new features, plus numerous bug fixes and speedups.
Incompatibilities
A top priority for new Kermit software releases has always been backwards
compatibility. A script written for a previous Kermit release should run
the same way in the new release.
There's one exception this time. The \fsplit() function is
incredibly handy, it can do almost anything, up to and including parsing a
LISP program (the underlying code is the basis of the S-Expression interpreter). But did you ever
try to use it to parse (say) a Tab-Separated-List (TSV file) or
Comma-Separated-List (CSV)? It works as expected as long as the data
contains only 7-bit characters. But if your data contains (say) Spanish or
German or Russian text written in an 8-bit character set such as ISO 8859-1,
every 8-bit character (any value 128-255) is treated as a break character.
This is fixed in C-Kermit 9.0 by treating all 8-bit bytes as "include"
characters rather than break characters, a total reversal of past behavior.
I don't think it will affect anyone though, because if this had happened to
anyone, I would have heard about it!
Since most standard 8-bit character sets have control characters in
positions 128-160, it might have made sense to keep 128-160 in the
break set, but with the proliferation of Microsoft Windows code pages, there
is no telling which 8-bit character is likely to be some kind of text, e.g.
“smart quotes” or East European or Turkish
accented letters.
What's Not In C-Kermit 9.0
Some large projects that were contemplated have not been done, including:
- IPv6. Honestly, there has been zero demand for this, and it
would be a lot of work and disruption to the code base. Volunteers
welcome, I guess. It could be a CS project.
- A database interface - MySQL or ODBC. For this one, there
is some demand but I haven't had a chance to even look into it
yet. I hope to include it in the final 9.0 release.
- There's a looming issue with DES encryption; major vendors are
removing it from their platforms, starting with Apple in Mac OS X 10.6, with
Microsoft to follow suit. A secure version of Kermit can be built without
DES, but in limited testing successful connections were spotty (e.g. with
Kerberos 5).
- Cleaning up the Unix makefile. It has 25 years' worth of targets
in it. It is very likely safe to remove most of them, since (a) most
old platforms have gone away by now, or have been upgraded, due to hacking
vulnerabilities; (b) the market has consolidated considerably; and
(c) most of the new features of C-Kermit 9.0, such as large files,
won't be of any use on older platforms and previous C-Kermit versions
will remain available.
- Packages. Everybody wants an install package custom made for
their own computer, Linux RPMs being the prime example but far from the only
one. These will come, I suppose (especially with some Linux sites having a
policy against installing any application that does not come as an RPM). In
the meantime, here's a page that describes some Kermit-specific issues in
package construction: ckpackages.html.
And a Loose
End...
Using External File-Transfer Protocols on Secure Connections
After C-Kermit 8.0.212 Dev.27 (2006/12/22), I spent a big chunk of time
trying to solve a particular problem that some of you have complained about
and others might be familiar with: If you use C-Kermit to make a secure
Telnet connection to another host (e.g. with Telnet SSL/TLS, Kerberos, or
SRP) and then attempt to transfer a file using an external protocol such as
Zmodem, it doesn't work.
That's because as coded (through 8.0.211), C-Kermit simply starts the
external protocol in a fork with its standard i/o redirected to the
connection. This completely bypasses the encryption and decryption that is
done by C-Kermit itself, and of course it doesn't work. The same thing
occurs if you use the REDIRECT command. The routine that handles this is
ttruncmd() in ckutio.c.
In order to allow (say) Zmodem transfers on secure connections, it is
necessary for C-Kermit to interpose itself between the external Zmodem
program and the connection, decrypting the incoming stream before feeding it
to Zmodem and encrypting Zmodem's output before sending out the connection.
In principal, this is simple enough. We open a pseudoterminal pair
("master" and "slave") for Zmodem's i/o and we create a fork and start Zmodem
in it; we read from the fork pty's standard output, encrypt, and send to the
net; we read from the net, decrypt, and write to the fork pty's standard
input.
In practice, it's not so simple. First of all, pseudoterminals (ptys) don't
seem to interface correctly with certain crucial APIs, at least not in the
OS's I have tried (Mac OS X, Linux, NetBSD, etc), such as select().
And i/o with the pty often – perhaps always – fails to indicate
errors when they occur; for example, when the fork has exited.
But, even after coding around the apparent uselessness of select()
for multiplexing pty and net, and using various tricks to detect when the
external protocol exits and what its exit status is, I'm still left with a
show-stopping problem: I just simply can not download (receive) a file with
Zmodem, which is the main thing that people would probably want to do. I
can send files just fine, but not receive. The incoming stream is delivered
to Zmodem (to the pty slave) but upon arrival at the Zmodem process itself,
pieces are always missing and/or corrupt. Yet I can receive files just fine
if I use Kermit itself (C-Kermit or G-Kermit) as the external protocol,
rather than Zmodem.
I can think of two reasons why this might be the case:
- Zmodem sends all 8-bit bytes and control codes in the clear,
and maybe the pty is choking on them because it thinks it is
a real terminal.
But Zmodem puts its controlling terminal into raw mode. And C-Kermit puts
the pty into raw mode too, just for good measure. If any 0xFF codes are in
the Zmodem data stream, and it's a Telnet session, Kermit does any needed
byte stuffing/unstuffing automatically. Anyway, if I tell Zmodem to prefix
everything, it makes no difference.
- Zmodem is a streaming protocol and perhaps the pty driver can't keep
up with a sustained stream of input at network speeds. What would be
the method of flow control?
I can vary the size of the i/o buffers used for writing to the pty, and get
different effects, but I am not able to get a clean download, no matter what
buffer size I use. write()'ing to the pty does not return an
error, and I can't see the errors because they happen on the master side.
It's as if the path between the pty slave and master lacks flow control; I
deliver a valid data stream to the pty slave and the master gets bits and
pieces. This impression is bolstered somewhat by the "man 7 pty" page in
HP-UX, which talks about some special modes for ptys that turn off all
termio processing and guarantee a flow-controlled reliable stream of bytes
in both directions – a feature that seems to be specific to HP-UX, and
exactly the one we need everywhere.
Well, in Pass One I used C-Kermit's existing pty routines from
ckupty.[ch], which are well-proven in terms of portability and
of actually working. They are currently used by SET HOST /PTY for making
terminal connections to external processes. But these routines are written
on the assumption that the pty is to be accessed interactively, and maybe
they are setting the fork/pty arrangement up in such a way that that's not
suitable for file transfer. The Pass One routine is called
xttptycmd() in ckutio.c.
So in Pass Two I made a second copy of the routine, yttptycmd(),
that manages the pty and fork itself, so all the code is in one place and
it's simple and understandable. But it still doesn't work for Zmodem
downloads. In this routine, I use openpty() to get the pty pair,
which is not portable, so I can have access to both the master and slave
pty file descriptors. This version can be used only a platforms that
have openpty(): Linux, Mac OS X, NetBSD, etc.
In Pass Three, zttptycmd(), I tried using pipes instead of
ptys, in case ptys are simply not up to this task (but that can't be true
because if I make a Telnet or SSH connection into a host, I can send files
to it with Zmodem, and the remote Zmodem receiver is, indeed, running on a
pty). But pipes didn't work either.
In Pass Four, I extracted the relevant routines into a standalone program
based on yttptycmd() (the openpty() version, for
simplicity), which I tested on Mac OS X, the idea being to rule out any
"environmental" effects of running inside the C-Kermit process. There was
no difference -- Kermit transfers (with C-Kermit itself as the external
protocol) worked; Zmodem transfers (neither sz or lsz) did not.
Well, it's a much longer story. As the external protocol, I've tried rzsz,
crzsz, and lrzsz. We know that some of these have quirks regarding standard
i/o, etc, which is one of the reasons for using ptys in the first place, and
i/o does work – just not reliably. Anyway, the 1100 lines
or so of ckc299.txt,
starting just below where it says "--- Dev.27 ---" tell
the full story. At this point I have to give up and move on; it
might be more productive to let somebody else who has more experience with
ptys take a look at it – if indeed anyone still cares about being able
to do Zmodem transfers over secure Telnet connections.
C-Kermit 9.0 contains the three new routines (and some auxiliary
ones), but they are not compiled or called unless you build it specially:
make targetname KFLAGS=-DXTTPTYCMD (builds with xttptycmd())
make targetname KFLAGS=-DYTTPTYCMD (builds with yttptycmd())
make targetname KFLAGS=-DZTTPTYCMD (builds with zttptycmd())
These are all in
ckutio.c. As
noted, the second one works only for Linux, FreeBSD, NetBSD, and Mac OS X,
because it uses non-POSIX, non-portable
openpty(). If you want to
try it on some other platform that has
openpty(), you can build it
like this:
make targetname "KFLAGS=-DYTTPTYCMD -DHAVE_OPENPTY"
(and let me know, so I can have
HAVE_OPENPTY predefined for that
platform too). The best strategy to get this working, I think, would be to
concentrate on
yttptycmd(), which is the simpler of the two
pty-based routines. If it can be made to work, then we'll see if we can
retrofit it to use the
ckupty.c routines so it will be portable to
non-BSD platforms.
By the way, if you build with any of [XYZ]TTPTYCMD defined, then
the selected routine will always be used in place of ttruncmd().
This is to allow testing on all kinds of connections, not just secure ones,
in both local and remote mode. Once the thing works, if it ever does, I'll
add the appropriate tests and/or commands.
By default, in the initial test release, C-Kermit 9.0 uses
ttruncmd() on serial connections and ttyptycmd() on
network connections. Even when a network connection is not encrypted,
Kermit still needs to handle the network protocol, e.g. the quoting of 0xff
bytes on Telnet connections.
– Frank da Cruz
fdc@columbia.edu