V10/cmd/netnews/README

Notes for 2.10:

There are some incompatiblities between 2.9 and 2.10.  None of them
should cause a problem, but you should be aware of them.  Some conversion
is needed.

The file storage format has been changed.  Rather than storing news in
$SPOOL/net.games.rogue/123, an article now goes in $SPOOL/net/games/rogue/123.
This permits newsgroup names to be longer than 14 characters and still
have subgroups.  It also makes directories smaller, resulting in faster
performance.  The dot files are gone, rather than saving the next
article number in $SPOOL/.net.games.rogue as the length of the file,
it goes in the active file on the same line as net.games.rogue.
Thus, your active file contains lines like
	net.games.rogue 00123
where the newsgroup name and the max article number are separated by
a space.  The article numbers are ALWAYS 5 digits long and include
the leading digits to do this (this is so they can be updated in place
without growing the active file).  The old active file has been saved
as $LIB/oactive in case of emergency.

This conversion of directory tree formats has an extra benefit.  You'll
find that readnews is now considerably faster than version 2.9.  The
movement of the dot files accounts for much of this, since it is no
longer necessary to "stat" every dot file.  Additionally, a routine to
find a newsgroup in your .newsrc has been modified to keep the
file sorted in the same order as active, and a "last found" pointer
is used to reduce the find time algorithm from quadratic complexity
(on the number of newsgroups) to linear complexity.  This makes the
total number of newsgroups less of a factor in determining speed,
and also keeps everyone's .newsrc cleaned out.  It is important
that people not store extra junk in their .newsrc, because it will
be deleted when they run readnews.

Before you do the conversion, you are urged to check your LIB/active
file and clean out any duplicates and old newsgroups.

Overall recommended conversion order:
	compile everything in this directory
		make
	make the new spool directories:
		sh cvt.mkdirs.sh /usr/lib/news /usr/spool/news
	convert the history file (non-USG only):
		cc cvt.hist.c -ldbm
		a.out /usr/lib/news
	convert the active file:
		sh cvt.active.sh /usr/lib/news /usr/spool/news
	install the inews binary:
		su
		rm -f /usr/bin/inews /usr/bin/rnews
		cp inews /usr/bin/inews
		chown news /usr/bin/inews
		chgrp news /usr/bin/inews
		chmod 6755 /usr/bin/inews
		ln /usr/bin/inews /usr/bin/rnews
		^D
	make links to spooled articles (this takes awhile):
		sh cvt.links.sh /usr/lib/news /usr/spool/news
		(see below)
	install the rest of the binaries:
		su
		make cp
		^D
	test everything to make sure it all works
	clean out the old spool hierarchy:
		sh cvt.clean.sh /usr/spool/news


Several shell scripts and C programs are provided to help with this
conversion.  The above scripts will convert active and create the new
directory tree.  (Links will be made to the old articles, so news can
continue to be read while you are doing this.)  Note that once you
install the inews binary, any incoming news processed by the new system
will not appear in the old system.  The above order was chosen to
minimize the window during which incoming news could be lost.  Once the
history and active files are converted, a new inews (and rnews) must be
installed, so that any incoming news will be kept in the new format.
Since the cross-linking is the time consuming part of the conversion,
it is deferred until after the critical window.  The commands above to
install inews are equivalent to "make cp" except that they only install
inews, not readnews or checknews.  This is so that anyone reading news
while you are cross-linking will be able to get at their news.
Otherwise, a "make cp" can be used if nobody will be reading news.  You
can save the old binaries in case of catastrophe, but converting back
once 2.10 has been running for any length of time is not recommended.

There are rumors of the cvt.links.sh part not working on a USG system.
If you have problems, the idea is to make a shell script with lines like
	ln net.news.group/* net/news/group
for each newsgroup, then to run the shell script.  An alternative
for USG is (roughly)
	sort /usr/lib/news/active -u -o /usr/tmp/dirs.2.9
	cat /usr/tmp/dirs.2.9 | sed -e 's;\.;/;' > /usr/tmp/dirs.2.10
	cd $SPOOL ; past /usr/tmp/dirs.2.9 /usr/tmp/dirs.2.10 |\
		awk '{print "ln "$1"/* "$2}' | sh
On an older system (e.g. V7) your ln command won't handle multiple
source files - you can create a shell script to do the same thing.
	
There are also rumors of cvt.mkdirs.sh not being able to make the
directories, for reasons nobody can figure out.  Being unable to
reproduce this, all we can offer is that if it happens to you,
you should do it by hand (an appropriate shell script can be made
easily with the editor.)  Please let us know if you figure out what
is going on.  It is normal to get "File exists" errors from mkdir
for local newsgroup (no dots in the name) and from parent newsgroups
(e.g. net.rec, net.bugs, net.sport, net.news) which you can safely ignore.

When you are satisfied that everything is working OK, run
	cvt.clean.sh $SPOOL
and it will remove the old dot files and directories.

The method used to determine if an incoming article has already been
seen locally has been changed.  On V7 systems, profiling showed that
rnews spent about 1/2 of its time in fgets reading the history file.
It now uses the dbm(3) library to hash the message ID of each article.
To avoid incompatibilities between 2.9 and 2.10, if you have more than
one incoming news feed, run the provided cvt.hist program, which will
enter all the message ID's from your 2.9 format history file into
the hashed database.  (On USG systems, there is no dbm library, but
fgets is much faster, so the problem should not be so pronounced.
inews 2.10 on a USG system will continue to use the old history file
method to detect duplicates.)  The text history file is maintained as
it was in 2.9, for human use.
	cc cvt.hist.c -ldbm
	a.out $LIB

Other incompatibilities:

The headers now meet the USENET format standard and RFC822.  Headers
stored will look verbose, and contain much more information.  Headers
transmitted to other systems will work with old B news systems or new
ones.  Once all your neighbors convert to the new standard, you can
take out the #define OLD in defs.h; this will prevent Posted and
Article-I.D. from being generated, saving some transmission costs
and making the stored articles a bit smaller.  The format of dates
has been changed to conform to RFC822.  If this causes your "Mail"
command to stop working with the -M option, it's easy to fix Mail.
In head.c, the first function "ishead" calls another function
"isdate".  Simply comment out the 4 lines calling isdate.

The user interface is roughly compatible, but the users will notice
a few differences, and there are a few extensions.  There are two
major differences in the interface.  The first is that postnews
now prompts for a "Distribution".  This defaults to the same as
the newsgroup (and will be omitted in this case), but allows the
user to conveniently enter a Distribution header line.  It also
makes them think about where their message is going to be sent
out to.  Any newsgroup(s) can be typed here, but normally you
just type either nothing, or a class distribution (e.g. "net",
"btl", "nj", "world"), restricting the distribution to that
class of sites.

The other change is that, on systems without page mode in the tty
driver (e.g. on systems with PAGE defined in defs.h), after printing
long articles with the pager, it will print a ":" and wait for
the user to hit return before printing the next header.  This
is to give the user a chance to read the whole article before
the next header forces it off the top of the screen.

Some less immediate interface changes:  Headers are displayed
in a format which is more compact but more information is displayed
than before.  If you want to see the article ID or full path,
or the date or newsgroups, the "h" command will show them.
The "H" command can be used for a full, verbose header dump.

Control messages are slightly different.  In particular, 2.10 now
requires that a newsgroup be created with a "newgroup" control message
(e.g. inews -C) before it can be used.  If an incoming article is in
some newsgroup that is not in the local active file, the article will
be locally stored in newsgroup "junk" and not forwarded to other systems.
This will prevent the accidental creation of typographical errors
by systems running older versions of news.  Also, the rmgroup control
message has been made less dangerous.  The defs.h parameter MANUALLY
is now defaulted to on.  This means that when an rmgroup message comes
around, you will be sent mail asking you to remove the group, but the
group will not actually be removed.  This prevents someone from
accidently or deliberately removing an important newsgroup such as
net.general.  If you control several machines and want to have
rmgroup messages automatically remove the group without your
intervention, disable MANUALLY.

expire used to automatically archive news in /usr/spool/oldnews if
that directory existed.  Now it won't archive it unless you give
it the -a option.

There is now a shell script called localize.sh which makes whatever
local mods you need to news, so you won't have to keep making the
same fixes for every release.  See localize.sample for an example
version, but you will want to fix it up to reflect your local
preferences.  Normally the affected files are Makefile and defs.h,
plus copying (with no changes) postnews, but you can make whatever
changes you need here.

CAUTION: If any of your neighbors run 2.9 or older news, you must
be sure you have OLD defined in your Makefile.  (The distributed
localize.sample will define OLD.)  Otherwise, your site will generate
articles with no Posted: or Article-I.D.: headers, resulting in
the "Inbound news is garbled" diagnostic at the remote end.

When you ship your news software to a new site, you must now send
a copy of your active file, so that the appropriate newsgroup
directories and active file can be built.  This was not necessary
in 2.9 because unrecognized incoming newsgroups were automatically
created, but they are thrown away in 2.10.

After the conversion, you should create two new newsgroups: junk
and control.  The first is for messages that are in an invalid
newsgroup (so you can see what's getting thrown away).  The second
is for control messages.  Ordinary users probably won't want to
read either, but as administrator you might want to.  To create
them, mkdir $SPOOL/junk $SPOOL/control, and add
	junk 00000
	control 00000
to the end of $LIB/active.

Note the addition of some new programs to $LIB.  These include
recmail (a program which takes a mail message on stdin, figures
out who the To: and Cc: lines refer to, and invokes /bin/mail
with those arguments [not changed] and with the file on stdin;
recmail is used by the reply command) and caesar (a program used
to decode rotated jokes; it can also be used to create rotated jokes).

Version 2.10 has some experimental batching provisions in it.
See the batch and unbatch programs, and the F and U options in
the sys file, for more details.  There are some related UUCP
modifications in the uucp directory that can also help performance
considerably.  All this is very new, and while it appears to run
OK, caution is advised in installing batching software.  You must
also make sure your neighbor is prepared to receive batched news;
this is normally true if the neighbor is running at least 2.10.

If you have an old C compiler which does not support the (void)
cast (V7 or C book based), you can add -Dvoid=int to CFLAGS in
the makefile to get the code to compile.