Newjunk: scanning the junk directory for new groups (Part 01/01)
Michael A. Meiszl
mam at mamnix.hamm.sub.org
Mon Feb 11 05:26:37 AEST 1991
The subject says it all :-)
For more information read the README file.
have fun
mam
#!/bin/sh
# This is a shell archive (shar 3.24)
# made 02/10/1991 18:22 UTC by mam at mamnix
# Source directory /usr2/mam/tools/newjunk
#
# existing files WILL be overwritten
#
# This shar contains:
# length mode name
# ------ ---------- ------------------------------------------
# 913 -rw-r--r-- Makefile
# 3761 -rw-r--r-- README
# 584 -rw-r--r-- newjunk.sh
# 7766 -rw-r--r-- newjunk.c
# 2030 -rw-r--r-- newjunk.h
# 107 -rw-r--r-- newjunk.msg
#
if touch 2>&1 | fgrep '[-amc]' > /dev/null
then TOUCH=touch
else TOUCH=true
fi
# ============= Makefile ==============
echo "x - extracting Makefile (Text)"
sed 's/^X//' << 'SHAR_EOF' > Makefile &&
X#
X# Makefile for NewJunk
X#
X
XCFLAGS=-Ox -Me3s -s
XCC=cc
XSHELL=/bin/sh
XBINDIR=/usr/lib/news
XOWNER= news
XGROUP= news
X
X
XTARGET= newjunk
XSCRIPT= $(TARGET).sh
XMSG= $(TARGET).msg
X
XOBJ= $(TARGET).o
XHDR= $(TARGET).h
XEXECUTABLE= $(BINDIR)/$(TARGET)
XEXESCRIPT= $(BINDIR)/$(SCRIPT)
XEXEMSG= $(BINDIR)/$(MSG)
X
X$(TARGET): $(OBJ) $(HDR)
X $(CC) $(CFLAGS) -o $(TARGET) $(OBJ)
X
Xclean:
X -rm $(OBJ) $(TARGET) a.out core
X
Xinstall: $(TARGET)
X -mkdir $(BINDIR)
X chown $(OWNER) $(BINDIR)
X chgrp $(GROUP) $(BINDIR)
X cp $(TARGET) $(EXECUTABLE)
X chown $(OWNER) $(EXECUTABLE)
X chgrp $(GROUP) $(EXECUTABLE)
X chmod 700 $(EXECUTABLE)
X cp $(SCRIPT) $(EXESCRIPT)
X chgrp $(GROUP) $(EXESCRIPT)
X chown $(OWNER) $(EXESCRIPT)
X chmod 700 $(EXESCRIPT)
X cp $(MSG) $(EXEMSG)
X chgrp $(GROUP) $(EXEMSG)
X chown $(OWNER) $(EXEMSG)
X chmod 600 $(EXEMSG)
X
Xshar: $(TARGET)
X shar -onewjunk -l50 Makefile README *.sh *.c *.h *.msg
X
X.c.o:
X $(CC) $(CFLAGS) -c $*.c
SHAR_EOF
$TOUCH -am 0210175091 Makefile &&
chmod 0644 Makefile ||
echo "restore of Makefile failed"
set `wc -c Makefile`;Wc_c=$1
if test "$Wc_c" != "913"; then
echo original size 913, current size $Wc_c
fi
# ============= README ==============
echo "x - extracting README (Text)"
sed 's/^X//' << 'SHAR_EOF' > README &&
X
X N e w j u n k
X -------------
X
X Install new newsgroups by scanning the junk directory.
X
XWritten 2/91 by Michael A. Meiszl (mam at hamm.sub.org)
X
XIf your systems receives news from more than one feed or from more than one
Xnet, you sometimes notice your junk directory growing pretty fast. This is
Xmostly due missing control messages from the nets. You can configure your
Xnews software to install a new group automatically on reception of a newgroup
Xcontrol message, but you can't do anything against a feed that decides to
Xsend you more groups without notice.
X
XSo its up to the system operator to check junk concorrently and to decide
Xwhether he needs to install that new group by hand or whether it was only
Xa crossposting from another net.
X
XManaging more than ones systems doubles the work of the operator and if this
Xhappens quite every day (we got big feeds here :-(), it was time for a cute
Xlittle program that does the work for me...
X
XNewjunk is a straightforward program which only reads through the junk direc-
Xtory and scans for the "Newsgroups:" line in the message header. It exctracts
Xthe wanted newsgroup(s) and checks against the active file. Since it only
Xreads the first field of the active file it will work with B- and C-news
Xwithout modifications. If the group is already there, nothing is done with it.
XThis keeps track of the chance when B-news will eject an valid article because
Xit is "too old".
XIf the group is not installed, newjunks checks if the group leader is already
Xthere. If not, the message is considered as a crossposting from another net
Xand the message is thrown away. If the net is there and the group is new,
Xnewjunk creates an call to inews gererating an newgroup control message for
Xa given distribution. Normally you set this distribution to "local", so only
Xyour system will be affected, but for managing more systems it is well suited
Xto create an own distribution, so newjunk must only be run on the main machine.
XAfter newjunk is done it can start a normal script file which is executed by
X/bin/sh. Look at my sample newjunk.sh for some ideas what you can get from it.
XThe tasks done by the script could easily be placed in the program itself,
Xbut a script is easier to adopt than fiddling with the code.
X
XINSTALLATION:
XEdit the Makefile newjunk.h and newjunks.sh for your needs then type "make".
XYou can edit the message in newjunk.msg too, but please leave the copyright
Xstatement in it.
XThis should build the executable in the local directory. You can test your
Xnstallation by calling "newjunk -t" which will do all scanning but no executing.
XIt will inform you about unreachable files or directorys and scan your junkdir.
XIf everything looks good, type "make install". This will copy newjunk and
Xnewjunk.sh into the news lib/bin directory and set permissions that it can
Xbe executed by the owner of the news system (normally "news").
XNow su to the news user and edit its crontab entry. Add a line for newjunk.
XRunning newjunk once a week or once a day depends on your traffic. Make
Xsure its run before your junk expires!
XYour'e done and hopefully are rid of some work too...
X
XCOPYING:
XThe program is put into the public domain, it may be copied and modificated
Xwithout notice as long as the original author's name is included. The
Xprogram may not be sold in any way. Especially semi-commercial "public-
Xdomain-resellers" are discouraged to keep and sell this program. Mailboxes
Xcan distribute it as long as they take no fee from there users.
XIf major modifications are made, please send me the diffs.
X
XBUGREPORTS:
X Send diffs an bugreports to:
X mam at mamnix.hamm.sub.org or mam at mamnix.garmhausen.de
X Michael A. Meiszl
X Tomsonstrasse 25
X D-W4700 Hamm 1
X Germany (the "West" is obsolete nowadays :-))
SHAR_EOF
$TOUCH -am 0210175391 README &&
chmod 0644 README ||
echo "restore of README failed"
set `wc -c README`;Wc_c=$1
if test "$Wc_c" != "3761"; then
echo original size 3761, current size $Wc_c
fi
# ============= newjunk.sh ==============
echo "x - extracting newjunk.sh (Text)"
sed 's/^X//' << 'SHAR_EOF' > newjunk.sh &&
X:
X# This is a sample application started from newjunk
X# This script is only executed if one or more groups were added by newjunk
X#
XBINDIR=/usr/lib/news
X#
XPATH=/bin:/usr/bin:/usr/local/bin:${BINDIR}
X#
Xcd ${BINDIR}
X#
X# First, let's sort our active file so that the new installed groups fit
X# neatly into the whole stuff
X#
Xsort -i -o active active
X#
X# Now spend some time and get rid of that unusable junk stuff
X#
Xexpire -e0 -E5 -n junk
X#
X# Last: save some diskspace (my history file is normally about 10mb...)
X#
X# NOTE: this works only with DBM history files!
Xrm -f ohistory oactive
SHAR_EOF
$TOUCH -am 0210175491 newjunk.sh &&
chmod 0644 newjunk.sh ||
echo "restore of newjunk.sh failed"
set `wc -c newjunk.sh`;Wc_c=$1
if test "$Wc_c" != "584"; then
echo original size 584, current size $Wc_c
fi
# ============= newjunk.c ==============
echo "x - extracting newjunk.c (Text)"
sed 's/^X//' << 'SHAR_EOF' > newjunk.c &&
X/*
X * newjunk written by Michael A. Meiszl (mam at mamnix.hamm.sub.org)
X */
X
X#include "newjunk.h"
X
X#define MAXITEMS 5000 /* if anybody cares for space lower it, but wouldn't
X it be better if you spend some bucks on rams ???
X */
X
Xchar *files[MAXITEMS]; /* Array for filenames,
X junk sometimes gets pretty big
X */
X
Xchar *groups[MAXITEMS]; /* Array for groups out of active file
X Currently I have 1154 groups online :-(((
X */
X
Xint maxfiles=0,maxgroups=0; /* Upper bounds of arrays */
X
Xint debug=0; /* Debug level. Currently only 9 does something
X special. Everything above 0 shows program
X progress
X */
X
Xint scriptflag=0; /* Incremented by every group that is newly
X installed. If a file argv[0].sh exists in
X the BIN directory it is executed if any
X group is created.
X */
X
X#ifdef HAVE_GETOPT
Xextern char *optarg;
X#endif
X/*
X * PANIC
X * Aborts program because of too low array configuration
X * Return: Never (I hope)
X */
X
Xvoid panic(what)
Xchar *what;
X{
Xfprintf(stderr,"Sysop! You have more than %d %s in your JUNK directory!\n",
XMAXITEMS,what);
Xfprintf(stderr,"Either you cleanup by hand or recompile newjunk with a higher MAXITEMS value!\n");
Xfprintf(stderr,"NEWJUNK: I give up!\n");
Xexit(2);
X}
X
X/*
X * USAGE
X * a nice little routine which only tells the user about his
X * stupidity :-)
X */
Xvoid usage()
X{
X printf("Usage: newjunk [-t] [-d NUM]\n");
X printf(" -t = testmode (show only, do nothing)\n");
X printf(" -d = debug level (9=testmode)\n");
X exit(0);
X}
X/*
X * READIN
X * reads the junk directory and copies filenames into the
X * files[] array.
X * Return: 0 if successful, -1 if open failiure
X */
Xint readin()
X{
X char l[255];
X DIR *D;
X struct direct *dir;
X
X if (D=opendir(JUNKDIR,"r")) {
X while (dir=readdir(D)) {
X if (*dir->d_name=='.') continue;
X files[maxfiles++]=strdup(dir->d_name);
X if (maxfiles>=MAXITEMS) panic("files");
X }
X closedir(D);
X return 0;
X }
X return -1;
X}
X
X/*
X * READACTIVE
X * reads the active file in and copies the groupnames to
X * the array groups[]
X * Returns: 0 if successful, -1 can't read active file
X */
Xint readactive()
X{
X FILE *F;
X char l[255],*s;
X
X if (!(F=fopen(ACTIVE,"r"))) return -1;
X while (fgets(l,255,F)) {
X for (s=l;*s && *s!=' ' && *s!='\t';s++);
X *s=(char) 0;
X groups[maxgroups++]=strdup(l);
X if (maxgroups>=MAXITEMS) panic("groups");
X }
X fclose(F);
X return 0;
X}
X
X/*
X * ADDGROUP
X * adds a new group to the array groups[] and calls INEWS to
X * create this group.
X * Returns: NOTHING or panics if array overflow
X */
Xaddgroup(which)
Xchar *which;
X{
X char l[255];
X FILE *P;
X
X if (maxgroups>=MAXITEMS) panic ("groups");
X
X groups[maxgroups++]=strdup(which);
X
X/*
X * build system() command line
X */
X strcpy (l,INEWS);
X#ifdef DISTRIBUTION
X strcat (l," -d ");
X strcat (l,DISTRIBUTION);
X#endif
X strcat (l," -C ");
X strcat (l,which);
X strcat (l," < ");
X strcat (l,BINDIR);
X strcat (l,"/");
X strcat (l,"newjunk.msg");
X
X if (debug) fprintf(stderr,"Executing:<%s>\n",l);
X
X/*
X * Execute command line with a read pipe
X * So we can get rid of inews output and keep pretty calm during crontab work
X * Why bother the sysop? He will get mail from the news system soon...
X */
X if (debug<9)
X if (P=popen(l,"r")) {
X while(fgets(l,255,P)) {
X if(debug) printf("INEWS:%s",l);
X }
X pclose(P);
X/*
X * Increment flag, so that MAIN can decide if it should start the script
X */
X scriptflag+=1;
X return 0;
X }
X else
X {
X perror("Can't Pipe to INEWS");
X exit(1);
X }
X else {
X fprintf(stderr,"no exec! just testing!\n");
X scriptflag+=1; /* fake it */
X }
X return 0;
X}
X
X/*
X * LOOKUP
X * searches the groups[] array for a given group
X * It is a poor linear scan, but who cares???
X * qsorting and bsearching maybe thousends of times faster,
X * but are not that portable as this routine
X * Returns: 0 if group has been added, -1 if group already exists
X */
Xlookup(what)
Xchar *what;
X{
X register int i,len;
X register char *s;
X
X/*
X * the stupid line parsing routine occasionally creates NULL strings, so
X * we eliminate them here
X */
X if (!strlen(what)) return -1;
X
X if (debug) fprintf (stderr,"searching for %s:",what);
X/*
X * first we scan for the whole group name
X */
X for (i=0;i<maxgroups;i++)
X if (!strcmp(groups[i],what)) {
X if (debug) fprintf (stderr," already there\n");
X return -1;
X }
X/*
X * Now, let's take a closer look to the net
X * If the first part of the group does not exist, don't add this group, it
X * may be a crossposting only.
X * SYSOP NOTE: Since this program will never create a new net, you have to
X * take care about that by yourself. If you don't bother, define NETCREATION
X * in the header file...
X */
X if (debug) fprintf (stderr," not there! checking net ");
X s=strchr(what,'.');
X if (!s) {
X if (debug) fprintf (stderr,": No TOP level allowed!\n");
X return -1; /* This would be a top-top level.
X I don't believe that...
X */
X }
X
X#ifndef NETCREATION
X *s=(char) 0;
X len=strlen(what);
X if (debug) fprintf(stderr,"%s:",what);
X *s=(char) '.'; /* in case we forget it later ... */
X for (i=0;i<maxgroups;i++)
X if (!strncmp(groups[i],what,len)) {
X if (debug) fprintf(stderr," ok, found %s\n",groups[i]);
X break;
X }
X if (i>=maxgroups) {
X if (debug) fprintf(stderr," new net, not allowed\n");
X return -1;
X }
X#endif
X/*
X * otherwise do your job...
X */
X if (debug) fprintf(stderr,"Installing new group %s\n",what);
X addgroup(what);
X return 0;
X}
X
X/*
X * SCAN_MESSAGE
X * reads the message specified as name and searches for the
X * "Newsgroups:" line. This line is broken up and fed into lookup.
X * Returns: 0 if ok, -1 if message is unreadable
X */
Xscan_message(name)
Xchar *name;
X{
X char l[255],*s,*t;
X FILE *F;
X
X sprintf(l,"%s/%s",JUNKDIR,name);
X if (!(F=fopen(l,"r"))) return -1;
X while (fgets(l,255,F)) {
X s=strrchr(l,'\n');
X if (s) *s=(char) 0;
X if (!strlen(l)) {
X if (debug) fprintf(stderr,"%s: no Newsgroups line!\n",name);
X break; /* End of Header */
X }
X if (!strncmp(l,"Newsgroups: ",12)) {
X s=l+12;
X/*
X * Pardon me this GOTO. It much more effective as strtok would be.
X */
Xredo:
X t=s;
X while (*t && *t!=' ' && *t!='\t' && *t!=',') t++;
X if (*t) {
X *t++=(char) 0;
X lookup(s);
X while (*t && (*t==' ' || *t=='\t' || *t==',')) t++;
X s=t;
X goto redo;
X }
X if (s) lookup(s);
X break;
X }
X }
X fclose(F);
X return 0;
X}
X
X
X/*
X * MAIN
X * nothing to worry about, it just calles the reading routines and
X * calls SCAN_MESSAGE for every entry of the files[] array
X *
X * Returns: I hope so :-)
X */
Xmain(argc,argv)
Xint argc;
Xchar *argv[];
X{
X int i;
X char script[255];
X
X debug=0;
X
X
X#ifdef HAVE_GETOPT
X
X while ((i=getopt(argc,argv,"td:v:x:"))!=EOF) {
X switch (i) {
X case 'd':
X case 'v':
X case 'x': debug=atoi(optarg); break;
X case 't': debug=9; break;
X default : usage();
X }
X }
X#else
X if (argc>1) debug=atoi(argv[1]); /* stupid, but reliable :-) */
X#endif
X
X
X/*
X * build filename for script
X */
X sprintf(script,"%s/%s.sh",BINDIR,argv[0]);
X
X if (debug) fprintf (stderr,"Reading JUNK directory: ");
X if (readin()) {
X perror("readin");
X exit(1);
X }
X if (debug) fprintf (stderr,"%d files found\n",maxfiles);
X if (debug) fprintf (stderr,"Reading ACTIVE file: ");
X if (readactive()) {
X perror("readactive");
X exit(1);
X }
X if (debug) fprintf (stderr,"%d groups found\n",maxgroups);
X for (i=0;i<maxfiles;i++) {
X if (debug) fprintf(stderr,"Scanning message %s\n",files[i]);
X scan_message(files[i]);
X }
X if (debug) fprintf(stderr,"%s: installed %d new group%s\n",
X argv[0],scriptflag,
X scriptflag>1?"s":"");
X if (scriptflag) {
X if (debug) fprintf(stderr,"%s: ready to start %s\n",
X argv[0],script);
X if (!access(script,1)) {
X if (debug!=9) i=system(script); else i=-1;
X if (debug) fprintf(stderr,"%s: exit status %d\n",
X argv[0],i);
X }
X }
X exit(0);
X}
X
SHAR_EOF
$TOUCH -am 0210174991 newjunk.c &&
chmod 0644 newjunk.c ||
echo "restore of newjunk.c failed"
set `wc -c newjunk.c`;Wc_c=$1
if test "$Wc_c" != "7766"; then
echo original size 7766, current size $Wc_c
fi
# ============= newjunk.h ==============
echo "x - extracting newjunk.h (Text)"
sed 's/^X//' << 'SHAR_EOF' > newjunk.h &&
X/*
X * NewJunk written 1991 by Michael A. Meiszl (mam at mamnix.hamm.sub.org)
X *
X * This version runs under SCO XENIX V2.3.3 but does nothing, any other
X * unix brand could not do either.
X *
X * The only thing you must take care about is where your header file for
X * the directory reading routines (opendir,readdir,closedir) is located.
X * Look under sys/ndir.h or dir.h or dirent.h.
X *
X * If you make major changes in the code, please send me the diffs.
X */
X
X/*
X * some defines you like to change
X */
X
X#define JUNKDIR "/usr/spool/news/junk" /* where the junk is located */
X#define INEWS "/usr/lib/news/inews" /* my favorite news software */
X
X/*
X * BINDIR tells the program where to look for its shell script
X * The script is named as the program but with a suffix ".sh"
X * normally its called "newjunk.sh"
X * If the script is ommited or not executable, no harm is done.
X * The script is mainly for tidying up and other stupid things your
X * system may require.
X */
X#define BINDIR "/usr/lib/news"
X
X/*
X * DISTRIBUTION tells the news software how far to spread the control
X * message. Please don't make it "world" you will certainly annoy some-
X * body. Keep it "local" as long as you don't manage more than one
X * system, otherwise make your own one (AND TELL IT TO YOUR NEWS-SOFT!)
X */
X/*#undef DISTRIBUTION /* don't want any, use default */
X/*#define DISTRIBUTION "local" /* normal case */
X#define DISTRIBUTION "mamnix" /* I have my own */
X
X/*
X * this is where the news software keeps its groups
X */
X
X#define ACTIVE "/usr/lib/news/active"
X
X/*
X * if you have getopt(s) define this
X */
X#define HAVE_GETOPT
X
X/*
X * if you like to create new nets automatically, define this
X */
X#undef NETCREATION
X
X#ifdef M_XENIX
X#define SYSV
X#endif
X
X#include <stdio.h>
X#ifdef SYSV
X/*
X * we need FILE and string definitions
X */
X#include <fcntl.h>
X#include <string.h>
X#else
X/*
X * I suppose BSD needs this one
X */
X#include <strings.h>
X#endif
X
X#include <sys/types.h>
X
X/*
X * this one is subject to be different on your system.
X */
X#include <sys/ndir.h>
SHAR_EOF
$TOUCH -am 0210171491 newjunk.h &&
chmod 0644 newjunk.h ||
echo "restore of newjunk.h failed"
set `wc -c newjunk.h`;Wc_c=$1
if test "$Wc_c" != "2030"; then
echo original size 2030, current size $Wc_c
fi
# ============= newjunk.msg ==============
echo "x - extracting newjunk.msg (Text)"
sed 's/^X//' << 'SHAR_EOF' > newjunk.msg &&
XThis group is automatically created by newjunk
Xwritten 2/91 by Michael A. Meiszl (mam at mamnix.hamm.sub.org)
SHAR_EOF
$TOUCH -am 0210175191 newjunk.msg &&
chmod 0644 newjunk.msg ||
echo "restore of newjunk.msg failed"
set `wc -c newjunk.msg`;Wc_c=$1
if test "$Wc_c" != "107"; then
echo original size 107, current size $Wc_c
fi
exit 0
--
Michael A. Meiszl, Tomsonstr. 25, 4700 Hamm 1 Internet: mam at hamm.sub.org
Voice: 02381/889121 Fax : 02381/880084 Fido : NO FIDO ANYMORE
DHST : 02381/880083 Traily: 02381/880081 Login : box | uucp | nuucp
More information about the Alt.sources
mailing list