UUCP proposal

[Ivan Shmakov]

[Cross-posting to news:news.misc, as netnews operation is being
discussed.]

Are there any specific reasons to be concerned with the reliance on
free news servers (that are run by the very same kind of amateurs we
are)?

What is the explanation that there are thousands of free fidonet
BBSes, but only a handful of free news servers?

Frankly, no idea. Perhaps a BBS is more fun to operate? One
can have an ASCII art logo there, as well as chat, games, etc.
-- something that a free news server would be lacking.

From there, it's trivial to apply any filtering one needs, whether
manual or automatic (so long as one can code the criteria, that is.)

I don't think filtering is what is required. I think groups should
be allowed to have a moderator, and the moderator should be able to
ask a feed to drop a user from the group. If the feed disagrees with
that, the feed should be able to split the group.

Wouldn't that feed then need a new moderator?

But I don't see why filtering cannot be used to solve that
problem. If the moderator decides to disallow posts originating
from a specific node, he or she can instruct other nodes to
discard messages with that node's name either at the tail or
anywhere in Path:.

This will still allow for netnews to be distributed in a mesh:
the group split can occur even if the group is being propagated
each-to-each among a group of nodes, some of which decide to
honor the moderator's decision, and some which don't.

Or that can be implemented at user level instead, much like was
the original idea behind NoCeM filtering.

There's X-Comment-To:, as used for netnews messages gated from
Fidonet (or Fidonet-style) Echomail.

Thanks! It would be good if that field allowed both a name and an
email address.

I suppose this field was intended to correspond to Echomail's
"To," and so there was no easy way for the format conversion
software to find an email address to put there.

It was pointed out before, however, that this field makes it
somewhat less clear that the intended recipient of Usenet posts is
in fact the group's readers.

I don't think it is less clear. Most messages are a reply to someone
else's message, but for group consumption. This just alerts you to
the fact that someone has replied to something you wrote.

A newsreader can readily obtain that information from its XOVER
cache. A separate field (and one which is /not/ by default
indexed in XOVER) isn't strictly necessary.

I don't see how it can make a difference for "casual users." Yes,
that may ease things somewhat for newsreader developers, but we
aren't expecting a boom of brand new software being developed to
read netnews (or echomail, for that matter), or are we?

I don't know about a "boom", but I tentatively wish to create or port
a public domain UUCP for PDOS/386. But I'm lacking a UUCP network to
hook into, not just the software. There's a Fidonet network, but you
can get kicked out of that, plus it doesn't use internet standards.

Unless I be mistaken, UUCP is not one of Internet standards, either.

Moreover, UUCP has its own disadvantages as compared to NNTP.
For instance, a UUCP node has to store all of the downstream
data until downloaded by its peers. If a peer goes down for
a month, that can accumulate to an amount that the operator may
decide isn't appropriate for them to keep. On the contrary,
NNTP requires only one copy of the data, from which any number
of downlinks can be fed.

Similarly, a node can efficiently obtain new articles for even
the same group from any number of peers over NNTP, while doing
so via UUCP would, in the worst case, mean retrieving essentially
the same data as many times as one has peers.

Also to note is that there's nothing specific in UUCP that netnews
relies upon: it's perfectly possible to use any other file
transfer protocol to distribute netnews "batches" -- be it Rsync,
HTTP, or indeed one of the Fidonet protocols (such as Binkp.)

As for the Internet protocols support, Contiki OS implements a
number of them and is available under a permissive free software
license (that is: one that not only puts no restrictions on the
/use/ of the code, but on its /distribution/ as well.)

(As far as I can tell, such permissive licenses tend to be
indistinguishable from "public domain" for all the possible
practical applications.)

https://en.wikipedia.org/wiki/FidoNet#Resurgence

W> The deaf and blind are also able to access this better than the
W> internet as a whole as interfaces for them deal mostly with ASCII
W> text which exists in most BBS. They find it helps them communicate
W> without the complications of pictures and audio in their internet
W> mail and communication in general.

That's misleading. First of all, ASCII in itself is no guarantee
of accessibility; one can easily thwart it
_ _ _ _ _ _
. (_) | _____ | |_| |__ (_)___
. | | |/ / _ \ | __| '_ \| / __|
. | | < __/ | |_| | | | \__ \_
._|_|_|\_\___| \__|_| |_|_|___(_)

Conversely, one can build accessible Web resources. I don't
have any document at hand outlining the design practices to
follow (writing such a reference is in my to-do list), but I use
Lynx as my primary Web reader, and as such can spot possible
accessibility issues, including when I write my own Web documents.

Also, there's a line-oriented (and thus friendly to both speech
synthesizers and Braille terminals) Edbrowse software.

I am interested in doing ASCII right. Oh yeah, I also want to port
UUCP to PDOS/3x0 and MVS, so that we can have BBS systems running on
EBCDIC. (But they would speak ASCII to the outside world).

Unless you're aiming at supporting anglophone users only,
I believe you should try to implement full Unicode support.

[Ivan Shmakov]

Basically I have more faith in hobbyist BBSes to carry traffic than
I do in hobbyist news servers.

I see no difference between the two.

I'd say there may be certain cultural differences.

I think BBSes are the right platform for free services.

The fundamental difference between USENET and an archaic bulletin
board is the issue of control. USENET is a distributed architecture
with no central point of control or storage. Very nice, that.
A BBS is no better than any random website which is completely under
control of the owner.

A BBS is no better than a random public news server, either, the
latter also being completely under control of the owner. However,
much like there're protocols in place to enable individual news
servers to exchange articles posted to common groups, there're
protocols to enable individual BBSes to exchange messages posted
to common boards. Which I suspect also tends to mitigate abuse
on the part of the operators.

[Ivan Shmakov]

What is the explanation that there are thousands of free fidonet
BBSes, but only a handful of free news servers?

Frankly, no idea. Perhaps a BBS is more fun to operate? One can
have an ASCII art logo there, as well as chat, games, etc. --
something that a free news server would be lacking.

Basically I have more faith in hobbyist BBSes to carry traffic than I
do in hobbyist news servers. I think BBSes are the right platform for
free services.

To me, "BBS" implies an "interactive-first" service; the one you
/have/ to have an active connection (be it via Internet, a phone
network, or some other medium) to use.

Sure, a BBS can provide QWK, UUCP, HTTP-based API, or some other
way to make use of its services in "batch" mode, but that's not
what I'd think BBS /must/ do.

Conversely, a node can offer QWK, UUCP, and other such services
without running any conventional BBS software whatsoever.

My personal preference is to use software of /my/ choice for
doing anything substantial, while possibly exchanging "batches"
of information with remote nodes ("hubs") as necessary to
propagate that information.

Which is something, say, Wikipedia has support for (the support
I've used myself to contribute to Wikipedia from Emacs -- in
place of a conventional Web browser), and which is something I
won't, in general, expect from a random BBS out there.

[...]

But I don't see why filtering cannot be used to solve that problem.
If the moderator decides to disallow posts originating from a
specific node, he or she can instruct other nodes to discard
messages with that node's name either at the tail or anywhere in
Path:.

I like the anarchic model of Fidonet where you can exchange messages
with whoever you want. At least if there are no rules about needing
to be in the nodelist.

While you can use Fidonet /software/ to exchange messages freely,
I believe that introducing a message from an excommunicated node
into the network proper would be against the Policy.

[...]

This will still allow for netnews to be distributed in a mesh: the
group split can occur even if the group is being propagated
each-to-each among a group of nodes, some of which decide to honor
the moderator's decision, and some which don't.
Or that can be implemented at user level instead, much like was the
original idea behind NoCeM filtering.

I believe the moderator should be able to ensure that messages from
the banee and any replies are prohibited, so it's not something that
can be done at the user level.

I'd say one can't really "ensure" that, but it's certainly
possible to consider replying to a banee's post as grounds for
banning the replying user oneself.

[...]

Similarly, a node can efficiently obtain new articles for even the
same group from any number of peers over NNTP, while doing so via
UUCP would, in the worst case, mean retrieving essentially the same
data as many times as one has peers.

I don't want to be reliant on an internet connection. The network
may instead be largely bluetooth connected in a Philippines village,
with ham radio being used for inter-village communication.

It's possible to operate a private IP-based network on top of any
bidirectional data stream; no connection to the global Internet
is ought to be necessary for the IP-based software to function
properly. That'd include a netnews server.

On the other hand, the NNTP protocol only requires a reliable
bidirectional data stream itself. (Not unlike, say, Zmodem.)
Hence, it's also possible to alter existent software to rely on
such a stream in place of TCP, or write a specialized NNTP
implementation for some such medium from scratch.

This is actually very simple: one can have multiple (packet)
networks (say, both wire and wireless); applying IPv6 (or IPv4)
on top of that yields the ability to forward (unreliably) a
packet from a node on one network to a node on any another; then
applying TCP on top of that stack gives reliable ordered streams.

Also to note is that there's nothing specific in UUCP that netnews
relies upon: it's perfectly possible to use any other file transfer
protocol to distribute netnews "batches" -- be it Rsync, HTTP, or
indeed one of the Fidonet protocols (such as Binkp.)

Ok. I actually already have a public domain zmodem that I wrote a
couple of decades ago. Maybe that can be used.

Upon reception of the file, one would then apply relevant
decompression algorithm (Gzip, Lzip, etc.) and run rnews(1)
(see, e. g., http://manpages.debian.org/rnews) to upload the
articles into a local NNTP server.

[...]

That's misleading. First of all, ASCII in itself is no guarantee of
accessibility; one can easily thwart it

I'm not catering for people who are trying to thwart the system.
I'm trying to have a solution for hobbyists to exchange mail,
including deafblind users.

Which can be a good reason to consider contributing to Edbrowse
development.

[...]

I am interested in doing ASCII right. Oh yeah, I also want to port
UUCP to PDOS/3x0 and MVS, so that we can have BBS systems running
on EBCDIC. (But they would speak ASCII to the outside world).

Unless you're aiming at supporting anglophone users only, I believe
you should try to implement full Unicode support.

I want to get it working for people who are using the US ASCII
keyboard, which would include Filipinos typing in Tagalog. I don't
believe the software should be burdened by people who refuse to use
an alphabet. They can use Pin Ying if they want.

And thus abandon the whatever little interlingual medium they
otherwise have. (AIUI, some Chinese "dialects" are as close to
each other as, say, modern English and French are. A common,
non-phonetic writing system reportedly aids communication greatly.)

Though the fact that tonal marks aren't ASCII themselves kind of
makes the point moot.

I would expect that Germans etc could use their extra characters too,
but there's no guarantee it will survive all the ASCII to EBCDIC
translations. But so long as the ASCII characters get translated to
the EBCDIC 1047 code page, that's all I'm trying to achieve.

As someone who deals with Cyrillic script on daily basis (and
also has a somewhat superficial interest in Japanese), I don't
think I'll have much interest in a pure-ASCII system. I certainly
wouldn't fancy explaining my colleagues that they have to put
effort into transcribing Russian in ASCII in order to communicate
with me. (And perhaps back, when they send me a text file and I
respond with comments /and corrections./)

You've quoted English Wikipedia earlier in this thread; wouldn't
it make sense for someone to quote Chinese Wikipedia in a
Chinese newsgroup? Would it really be a good idea to require
the poster to transcribe the quote into Latin script first?

Also, my focus is on alt.os.development where communication is in
English even though we have non-Anglophones present. I'd like to
focus on getting that group onto thousands of BBS systems instead of
what I am currently doing which is to read/write via Google Groups,
and having no archive on my own machine.

That's something I find puzzling as well: why do you still use
Google Groups when it's quite easy to set up INN and a newsreader
and use that instead?

And I want to do it on PDOS/386, PDOS/380 and MVS/380 too.

Using a newsreader (and a public news server) running on
whatever OS you currently use would already be an improvement
over using public Google Groups via a Web browser running on
that same OS, IMO.

[Ivan Shmakov]

Post by Ivan Shmakov

I like the anarchic model of Fidonet where you can exchange
messages with whoever you want. At least if there are no rules
about needing to be in the nodelist.

While you can use Fidonet /software/ to exchange messages freely,
I believe that introducing a message from an excommunicated node
into the network proper would be against the Policy.

1. Use internet technology instead of Fidonet technology.
2. Does not have a policy document.

This sounds backwards: the policy document does not /cause/
conflicts; rather, it provides guidelines for their resolution
(peaceful or not.)

While the specific guidelines that comprise the Fidonet Policy
may be good or bad, merely getting rid of that document will in
no way make conflicts impossible.

I wish to communicate in English with the people in alt.os.development,
and also communicate with deafblind users in the 3rd world who are
using Morse code vibrators to read English messages.

I believe you're underestimating the cost of deployment of said
vibrators.

Post by Ivan Shmakov
I'd say one can't really "ensure" that, but it's certainly possible
to consider replying to a banee's post as grounds for banning the
replying user oneself.

The person doing the replying may not know that the original person
was banned, and I don't want to see such people also banned.

In that case, it's likely that the user who replied does not
recognize the authority of that moderator anyway, effectively
being on the other side of the "split."

If that's a new user, however, the moderator can issue a warning
instead of banning him or her outright. It will still be up to
the user to decide if he wants to heed that warning or not.

[...]

Post by Ivan Shmakov
On the other hand, the NNTP protocol only requires a reliable
bidirectional data stream itself. (Not unlike, say, Zmodem.)
Hence, it's also possible to alter existent software to rely on such
a stream in place of TCP, or write a specialized NNTP implementation
for some such medium from scratch.

Ok. But I think an efficient system should be based around compressed
batches like BBSes do, and if some nodes have bandwidth to spare, it
should be an additional thing, not a replacement for the efficient system.

The Golden Rule of Networking is that by saving bandwidth you
waste cycles. As such, both of the approaches /are/ efficient:
compressed batches save bandwidth, while polling may save cycles.
Whether one or the other are at surplus may vary from user to user.

But I'd like to point out that Transport Level Security (TLS)
protocols provide support for transparent compression, even though
their focus is transparent /encryption./ Then again, the latter
will likely be desirable for a wireless link anyway.

AIUI, in "streaming" mode, NNTP may be rather friendly to
compressed streams. There, the client will first send a
compressed "packet" containing commands to check the groups of
interest; based on the server's compressed response, the client
would then construct a "packet" of XOVER data requests; and the
response to that would be used to request the articles themselves
-- which would at that point be compressed and sent by the server.

Post by Ivan Shmakov

I want to get it working for people who are using the US ASCII
keyboard, which would include Filipinos typing in Tagalog. I don't
believe the software should be burdened by people who refuse to use
an alphabet. They can use Pin Ying if they want.

And thus abandon the whatever little interlingual medium they
otherwise have. (AIUI, some Chinese "dialects" are as close to each
other as, say, modern English and French are. A common,
non-phonetic writing system reportedly aids communication greatly.)

I don't think software should be burdened with non-alphabets unless
we're willing to do the same with English words, and have a "Eurocode"
which encodes entire words in the European languages, instead of just
characters.

There were some attempts at that in the past; see, e. g., [1].

The obvious difference of such a system to Chinese characters
would be that the latter has a sheer volume of existing literature,
that you may want, or sometimes need, to read and quote.

[1] http://en.wikipedia.org/wiki/Pasigraphy

[...]

Post by Ivan Shmakov
That's something I find puzzling as well: why do you still use
Google Groups when it's quite easy to set up INN and a newsreader
and use that instead?

Such software doesn't come pre-installed on Windows so I need to
investigate that. Nor did I have information about free news
servers.
Using google groups is the equivalent of logging on to a BBS and
using the BBS system itself to interactively write a message. That's
the normal introduction to messaging. After that you start asking
"how can I do this in bulk?".
I used to run a BBS call the "Ten Minute Limit" which had a limit of
10 minutes connection time, and only one file area visible, which
contained the software required to download and do FREQs to get files
or pointing software to become a point.
I am very satisfied with such a setup. Google Groups doesn't have
such a thing I can click on.

Yet you can use Google /Search/ to find Alpine, or Thunderbird
(or NeoMutt, Gnus/Emacs, etc.), download one and point it at
nntp://news.aioe.org/alt.os.development.

I admit that it was easier to me: the GNU variant I've used when
I began to use Internet /did/ come with both several newsreaders
and a news server software installable "in a single click."
(Fidonet connectivity software was included as well.)

Hence, and as my dial-up Internet connection was paid by minute,
it made every sense for me to set up a local NNTP server and only
connect to Internet for as long as necessary to bulk-download the
new articles for off-line reading. (And to upload my followups.)

I'm afraid I've never investigated the possibility of running INN
(and either NewsStar or Suck) on a non-GNU system. Well, except
for when my Fidonet uplink decided to move to FreeBSD and I've
helped him to set up most of the relevant software.

[...]

[Ivan Shmakov]

Post by Ivan Shmakov

2. Does not have a policy document.

This sounds backwards: the policy document does not /cause/
conflicts; rather, it provides guidelines for their resolution
(peaceful or not.)
While the specific guidelines that comprise the Fidonet Policy may
be good or bad, merely getting rid of that document will in no way
make conflicts impossible.

I am expecting conflicts. And when they occur I expect people to
stop exchanging mail with people they don't like, rather than someone
getting excommunicated.

The issue at hand is that of transit data. Suppose, for
example, that Alice lives at a remote location and her only
access to the network is via a radio link with Bob. Now, Bob
has a conflict with Dave, and stops exchanging data with him.

From the Alice's viewpoint, Bob just got Dave excommunicated.

The only "real solution" to this problem I know of is to have
all transit data end-to-end encrypted (as, e. g., GNUnet does.)
When Bob only knows that the packet he's got from Charlie is to
be forwarded to Alice, he's likely to have no qualms with doing
so, even if the packet in fact ultimately originates from Dave.

After all, I am free to exchange snail mail with my friends too.
There's no authority with the power to stop me from communicating
with my friends. By telephone too.

Unless "national security" gets involved, that is.

Post by Ivan Shmakov

I wish to communicate in English with the people in alt.os.development,
and also communicate with deafblind users in the 3rd world who are
using Morse code vibrators to read English messages.

I believe you're underestimating the cost of deployment of said
vibrators.

Regardless of the challenges, it's cheaper than Braille readers,

There's also the challenge of making Braille terminals cheaper.

Back when I've investigated the issue, my understanding was that
a huge part of the cost is due to the piezoelectric actuators
employed. (Which a single device needs around a few hundreds;
though given that my first computer used a 12-or-so-character
display, and was /not/ unusable at that, I suppose around one
hundred actuators may be enough, provided that scrolling is
somehow made easy.)

and it is an abstraction I find interesting anyway - being able to
communicate in English via Morse Code.

Well, if anything, I can suggest to put effort to make the
design manufacturable. Then you can "leak" the files to Chinese
manufacturers, and there's at least some chance that the
(arguably limited) market will get flooded with cheap "knockoffs."

[...]

Post by Ivan Shmakov
The Golden Rule of Networking is that by saving bandwidth you waste
cycles. As such, both of the approaches /are/ efficient: compressed
batches save bandwidth, while polling may save cycles. Whether one
or the other are at surplus may vary from user to user.

Ok, I wish to focus on saving bandwidth. I don't think the CPU
requirements for compressing and decompressing are an issue.
Also note that some of the links may be a physical USB stick carrying
data. I want minimum time spent offloading data.

If you consider CPU cycles to be cheap, you can form the batch
the moment you get the USB flash device mounted -- instead of
making them whenever there's new data.

So, in place of having a configuration file /on your host/ that
defines which messages should be fed to batches for a specific
downlink, you'd have the same, or similar, 'request file' that
the interested person would bring to you on the flash device
itself. From where it'd be possible for your downlink to
request for /any/ messages that you still keep at the time of
retrieval (and not just the newly received ones.)

Consider, for instance this scenario: the person regularly gets
messages posted to the foo.bar group from you, and at one point
learns that there was a recent discussion on baz.qux of interest
to him. He then updates the request file so that it reads "get
also the messages which were posted to baz.qux in the last three
weeks." Upon the next exchange, he gets the relevant portion of
your archive.

Another scenario is that you regularly form batches for some
person for months, only to discover that he's lost interest in
the groups you carry in the mean time.

Post by Ivan Shmakov
The obvious difference of such a system to Chinese characters would
be that the latter has a sheer volume of existing literature, that
you may want, or sometimes need, to read and quote.

That is not the problem I am trying to solve at the moment. I am
focused on English language forums being accessible to anyone who
speaks English, regardless of whether they are deafblind in the 3rd
world, or whether they are using EBCDIC platforms.

That's part of my point: how the number of users that you expect
to be interested in EBCDIC support compares to the number of
those who'd want Chinese instead?

[...]

Post by Ivan Shmakov
Yet you can use Google /Search/ to find Alpine, or Thunderbird (or
NeoMutt, Gnus/Emacs, etc.), download one and point it at
nntp://news.aioe.org/alt.os.development.

Ok, yes, I agree it is possible.
But I intend to make my BBS do it the way I think is best. Have a
single file area with software required to do an anonymous UUCP to
get access to more software and for some of that software to be a
UUCP news reader.

I have to point out that it's currently considered a good
practice to only install software originating from a "trusted"
source. For instance, about the only source I trust nowadays is
the Debian project; thus virtually any program that I find
elsewhere I'll only run in an "isolated" environment (such as
under QEMU, or on a separate "diskless" machine -- running from
a dedicated USB flash drive that I'll readily "reformat" after
the particular experiment is over.)

I'm not familiar with that currently. I'm only familiar with
Fidonet, as I worked on the software there.

Please note that I'm /not/ arguing that you should use, say,
NeoMutt and Aioe in preference to your own BBS software. Rather,
my point is that you should rely on NeoMutt/Aioe instead of
Google Groups.

That will allow you to familiarize yourself with Usenet concepts
(because Google Groups is /not/ a Usenet user agent, but rather
a Google's own Web forum, which also happens to include a
Usenet-to-Google bidirectional gateway) and existing software
(whose features you may decide to reimplement in your own.)

Moreover, that way you'd be using a hobbyist's server instead of
a "free" service run by a for-profit corporation, which I gather
is something you'd prefer anyway.

(As for the "Web forum" part, please consider that I'm filing my
posts in this thread under both "alt.os.development" and
"news.misc." A conventional netnews user agent would by default
file followups likewise. However, as Google forum does /not/
support crossposting, your responses are instead only filed
under "alt.os.development.")

Also hopefully the userid/password created for the BBS can be used
for UUCP news.

That's certainly doable in principle.