back to index

COM over TCP/IP, RFC2217


Serial port over network - COM/UART/tty over TCP/IP, RFC2217
      Serial port/cable hardware
      Signals, by pin
            input:Rx, output:Tx
            output:RTS, Request To Send
            output:DTR, Data Terminal Ready
            input:DCD, Data Carrier Detect; also CAR, Carrier; CD, Carrier Detect; RLSD, Receive Line Signal Detect
            input:DSR, Data Set Ready; also LE, Line Enable
            input:CTS, Clear To Send
            input:RI, Ring Indicator; also RNG, Ring
            reference:GND
      RS232 - DTE, DCE, null-modems
      Wiring and signals, by use
            modem
            null-modem
            mice and assorted small serial devices
            digital control signals (Rx/Tx unused)
            GPS
            bootloaders
            RS485
            UART
      Drivers and levels
            mark/space/break
            UART
            RS232
            RS422
            RS485
            old terminals/teletypes
Serial port, software interfacing
      serial port configuration
            data frame
            break
            baudrate vs bitrate
            speed constraints
            flow control
            split speeds
            speeds
      ioctl calls and structures
            c_cflag, control mode flags - speed, parity, bits
            c_iflag, input mode flags - input data handling/translations
            c_oflag, output mode flags - output data handling/translations
            c_lflag, line discipline flags - terminal control, control characters processing
            c_cc[] array, control characters
            terminal-initiated signals (when ISIG enabled)
Serial-over-TCP/IP, RFC-2217
      RFC 854, other telnet RFCs
      RFC 2217
            Control sequences, telnet console examples
            Control sequences, serial port examples
Serial port over network
      linux servers
            rfc2217_server.py - good
            ser2net - good?
            remserial - bad (no RFC2217)
      linux client
            ttynvt - good
            remserial - bad (no ioctls)
            socat - bad (no ioctls?)
            socat-rfc2217 (todo: check)
      windows client
            hercules terminal
            HW VSP 3
            Free Virtual Serial Ports
ttynvt modifications
      possible future expansion
      issues
            esptool.py
                  crude solution:
      download

Serial port over network - COM/UART/tty over TCP/IP, RFC2217

Serial port/cable hardware

Many devices are interfaced via a form of a serial port. The possible implementations are:

Many microcontrollers, from Arduino-class to ESP8266 and ESP32, are equipped with a serial port and a bootloader, allowing in-system flashing of new firmware.

Signals, by pin

input/output from the perspective of the computer ("DTE")

input:Rx, output:Tx

output:RTS, Request To Send

output:DTR, Data Terminal Ready

input:DCD, Data Carrier Detect; also CAR, Carrier; CD, Carrier Detect; RLSD, Receive Line Signal Detect

input:DSR, Data Set Ready; also LE, Line Enable

input:CTS, Clear To Send

input:RI, Ring Indicator; also RNG, Ring

reference:GND

RS232 - DTE, DCE, null-modems

The serial output has two data lines:

RS-232 devices have two flavors:

RS-232 connectors come in a few common variants (and a plethora of ad-hoc vendor-specific ones):

EIA signal 9pin  25pin RJ45   DTE (DCE)  pairs
BB  Rx      2      3    5     IN   out    Tx     serial data receiving
BA  Tx      3      2    6     OUT  in     Rx     serial data transmitting
CA  RTS     7      4    8     OUT  in     CTS    computer asks modem to send data
CB  CTS     8      5    7     IN   out    RTS    modem is ready to accept data - response to RTS, after few msec
CD  DTR     4     20    3     OUT  in  DCD+DSR   computer is operational; when asserted, accepts/continues call; often low by OS when port is unused
CC  DSR     6      6   (1)    IN   out    DTR    modem is operational
CF  DCD     1      8    2     IN   out    DTR    modem indicates carrier present, remote side is sending signal"; presence of remote party and active connection
CE  RI      9     22    1     in   out    n/a    modem announces incoming call; input on computer, omitted on nullmodems
AB  GND/SG  5      7    4     ref  ref           signal ground
AA  PGND           1                             protective ground, shielding

Wiring and signals, by use

modem

null-modem

ALWAYS: A:Rx-----B:Tx, A:Tx-----B:Rx, A:GND----B:GND simple: no additional lines self-control: A:RTS++A:CTS, B:RTS++B:CTS, A:DTR++A:DSR++A:DCD, B:DTR++B:DSR++B:DCD part-self: A:DTR++A:DSR++A:DCD, B:DTR++B:DSR++B:DCD RTS-CTS: A:RTS----B:CTS, A:CTS----B:RTS partial: A:RTS----B:CTS, A:CTS----B:RTS; A:DTR++A:DSR++A:DCD, B:DTR++B:DSR++B:DCD EYN253C,full: A:RTS----B:CTS, A:CTS----B:RTS, A:DTR----B:DSR++B:DCD, B:DTR----A:DSR++A:DCD EYN254C: A:DTR----B:DSR++B:CTS++B:DCD, B:DTR----A:DSR++A:CTS++A:DCD

mice and assorted small serial devices

digital control signals (Rx/Tx unused)

GPS

bootloaders

RS485

UART

CAUTION: as demonstrated, there is no pinout standard for board UART connectors!

Drivers and levels

mark/space/break

UART

RS232

RS422

RS485

old terminals/teletypes


Serial port, software interfacing

In Linux, the serial ports are character devices. Plain input/output is handled by writing data to the device and reading from it.

"Out of band" operations, like setting baud rate and handling the control lines, is done by performing ioctl operations on the opened file.

The serial port names are usually:

Windows have a very similar concept to ioctl, the DeviceIoControl call.

serial port configuration

For the communication, several configuration options are required:

data frame

for 8N1 (8bits, no parity, 1 stop bit) there are 10 bits per byte transmitted; 9600 bps == 960 bytes/sec

A sequence of one lonely byte, compared with a few together, looks

For 8N1 (most common):
 ^^^^^^^^sddddddddS^^^^^^^^
 ^^^^^^^^sddddddddSsddddddddSsddddddddSsddddddddS^^^^^^^^^ structure
 111111110dddddddd10dddddddd10dddddddd10dddddddd1111111111 template
 111111110001010101010100110101100111010001011101111111111 example 8N1
          00101010  10100110  11001110  00101110           payload

For 7E1 or 7O1:
 ^^^^^^^^sdddddddpS^^^^^^^^
 ^^^^^^^^sdddddddpSsdddddddpSsdddddddpSsdddddddpS^^^^^^^^^ structure
 111111110dddddddp10dddddddp10dddddddp10dddddddp1111111111 template
 111111110001010111010100110101100111110001011101111111111 example 7E1
 111111110001010101010100111101100111010001011111111111111 example 7O1
          0010101   1010011   1100111   0010111            payload

For 8E1 or 8O1:
 ^^^^^^^^sddddddddpS^^^^^^^
 ^^^^^^^^sddddddddpSsddddddddpSsddddddddpSsddddddddpS^^^^^ structure
 111111110ddddddddp10ddddddddp10ddddddddp10ddddddddp111111 template
 111111110001010101101010011001011001110110001011100111111 example 8E1
 111111110001010100101010011011011001110010001011101111111 example 8O1
          00101010   10100110   11001110   00101110        payload

For ol' 5N2/ITA2:
 ^^^^^^^^sdddddSS^^^^^^^^^^
 ^^^^^^^^sdddddSSsdddddSSsdddddSSsdddddSS^^^^^^^^^^^^^^^^^ structure
 111111110ddddd110ddddd110ddddd110ddddd1111111111111111111 template
 111111110000011101000011010100110000011111111111111111111 example 5N2
          00001   10000   10100   00001                    payload

  ^ - idle line - UART in H
  s - start bit - 1 cycle, L
  d - data bits - usually little-endian, LSB-first
  p - parity bit - optional, last in data bit sequence
  S - stop bit - 1, 1.5 or 2 cycles, H; functionally equivalent to idle line

  test sequence: ASCII: "Test", 0x54-0x65-0x73-0x74
                 ITA2:  "TEST", 0x10-0x01-0x05-0x10 ("Baudot")

There may or may not be additional arbitrary space (not necessarily a multiple of bit cycle) between the stop bit and the subsequent start bit. The front edge of the start bit synchronizes the receiver's bit clock.

break

A break is a stream of zero bits, usually for 250-500 msec. Usually either

Sent by IOCTLs:

baudrate vs bitrate

In communication, symbols are fed through the data line at a given rate per second. A symbol can be a discrete value of signal amplitude (or voltage on the line), frequency or frequency combination (FSK - a discrete-level variant of FM), phase (PSK), or combination (QAM, QPSK...)...

There are possible encodings of bits per symbol, ranging from one (or less than one in some cases) to many:

type        bits/baud   type
logical 1/0      1      common serial UART
QPSK             2
8-PSK            3
16-QAM           4
64-QAM           6
256-QAM          8
CIS-45          45      OFDM, 45 parallel tones, 33.33 or 40 Bd (1500 or 1800 bps)
CIS-60          60      OFDM, 60 parallel tones, 35.555 Bd (2133.33 bps)
CIS-93          93      OFDM, 93 parallel tones, 22 Bd (2046 bps)

speed constraints

flow control

To give the receiving system the ability to tell the sender to back off a little because it is sending data too fast, flow control schemes can be deployed. The most common are:

split speeds

In many cases the speed needed in one direction is much lower than for the other direction.

A terminal attached to a computer can benefit from high speed on the downlink but the same speed would be wasted on the meat-in-the-chair's keystrokes.

Hence, split speeds (downlink/uplink); eg.:

Similar principle applies for eg. ADSL lines, where the assumption is that the downlink bandwidth (web browsers, video, assorted downloads) will be needed to be way higher than the uplink (webpage requests, occasional email or chat message).

speeds

The speed of the line, the bitrate (or baudrate, for usual UART/RSxxx/currentloop one-bit-per-baud they are equal), can vary widely.

The bit clock is generated by binary division of a master clock source, usually a crystal oscillator. (PLL can be also employed but these are rare with ordinary serial ports.)

Many devices have fixed list of supported bitrates. Some have the unused once-common-standard speeds remapped to more modern ones missing in the tables. Eg. some Amiga computers have the 31250 bps rate (common for MIDI) in the setting for 134 bps - the target audience had more musical equipment to talk with than IBM Selectrics.

 c_flag = bitrate identification number for unix termios structure
 DIV = divisor from 1.8432 MHz clock divided by 16 = from 115200
 DIV1 = divisor from 1 MHz
 DIV12 = divisor from 12 MHz (FTDI chips use 3MHz clock reference)
 CAN = CAN Bus
 NICAN = National Instruments NI-CAN hardware - CAN bus bitrate, less common

 s=standard speed
 S=standard speed, common
 c=CAN bus speed
 C=CAN bus speed, common
 r=radio link speed
 t=teletypes
 T=teletypes, common
 m=modems
 i=industrial

            rate   c_cflag  DIV DIV1 DIV12
           OTHER   0x1000                   requires termios2, the speed is then in c_ispeed, c_ospeed
               0   0x0000                   no connection
tr            45                            RTTY 45 baud, common amateur standard, 5-bit (eg. 14075 KHz USB); often a shorthand for 45.45(45)
Tr       45.4545               264000  TTY V.18, "60-speed" teleprinters (USA), 5-bit/ITA2, 5N2; amateur-radio RTTY; Teletype Model 15, most audio recording of teleprinters
tr            50   0x0001   2304    240000  TTY V.18, "66-speed" teleprinters (Europe), 5-bit/ITA2, 5N2; common Telex, news agency wires, from year 194x
tr            56.875                        "75-speed" teleprinters, 5-bit/ITA2
tr            74.2                          "100-speed" teleprinters, 5-bit/ITA2
Tr s          75   0x0002   1536    160000  5-bit teleprinters; reverse channel in some V.23 1200bps modems; RTTY 75 baud, 5 bit (eg. 10536 kHz USB), common standard in weather data transfer
 r           100           10000    120000  packet radio, AMTOR
T ms         110   0x0003                   typical teleprinter speed; Bell 101 modem
T            134.5 0x0004                   IBM2731 mechanical teletype terminal (1.5 stop bit); IBM Selectric typewriter - at 110 bps the internal clutch kept engaging/disengaging and wearing out mechanism[ref]
  ms         150   0x0005    768     80000  2*75; acoustic coupler modems, typical reliable speed
 r           200   0x0006    576     60000  some terminals; unusual; packet radio, PACTOR
  ms         300   0x0007    384     40000  modem, V.21 (full duplex, 300baud, FSK); acoustic coupler modems, 1972; Pennywhistle modem; Hayes Smartmodem (audio FSK, 1981), Bell 103 modem (also packet radio, eg. 14105 kHz USB, 147.060 NFM); "300bps,N,8,1"; common packet radio speed below 30 MHz
             450             256            some terminals; unusual
 r           512             225            POCSAG 512 baud, radio, paging format originating in the UK, used widely in the US
  ms         600   0x0008    192     20000  modem, V.22 (full duplex, 600baud, PSK); Golay Motorola radio paging signal
  ms        1200   0x0009     96     10000  modem, V.22 (full duplex, 600baud, QPSK), Bell212A (QAM), Bell202 (1200baud, FSK), V.23 (half-duplex, FSK, sometimes 75bps reverse channel, eg. French Minitel, German BTX); acoustic coupler practical upper limit; Bell 202 modem AFSK (1200Hz mark, 2200Hz space, half-duplex; packet radio (eg. 144.39 NFM, 145.030 NFM), HART); DALI[ref] bus; POCSAG 1200 baud; various telemetry
  m         1600                            FLEX, radio paging (929 and 931 MHz US)
  m         1800   0x000a     64            less common; some leased-line modems, Bell 202C/D/R/T (FSK); about the ceiling of FSK on phone line; CP2101
            2000                      6000  some terminals; unusual
! mS        2400   0x000b     48      5000  modem, V.22bis (full duplex, 600baud, QAM), V.26bis (1200baud, PSK); POCSAG 2400 baud; DPSK common modem modulation; some multimeters
 rm         3200                            FLEX, radio paging (929 and 931 MHz US)
 rm         3600              32            0.5*7200; modem, Bell 203A/B/C (vestigial sideband); unusual; various mobile radio trunking control (Motorola Type 1)
            4000                  250 3000  unusual; CP2102
!rmS        4800   0x000c     24      2500  modem, V.32 (half-duplex, 2400baud), V.29 (PSK/QAM), V.32bis, V.27ter (1600 baud, PSK, 1650Hz carrier), Bell 208A/B (8phi-PSK); wireless, dispatch control systems, Automatic Train Control System; some slower GPS; various mobile radio trunking control
    c       5000                  200 2400  NICAN
    c       6150                            NICAN
 r          6250                            ERMES paging (European Radio MEssage System), wireless (169MHz in France, Hungary, Malaysia)
 r          6400                            FLEX, radio paging (929 and 931 MHz US)
 rm         7200              16            modem, V.29 (PSK/QAM), V.32bis, Bell 203A/B/C (vestigial sideband); unusual; CP2101
    c       7812.5                128 1536  NICAN
    c       8000                  125 1500  NICAN
!rmS  i     9600   0x000d     12      1250  modem, V.32 (half-duplex, 2400baud, QAM, 1650Hz carrier), V.29 (PSK/QAM), V.32bis, Bell209A (QAM); common factory default (8N1) for slower speeds; very common general communication rate for lower-speed devices (GPS, multimeters, MODBUS, Profibus...); various mobile radio trunking control
    c      10000                  100 1200  NICAN
           11520              10            1.2*9600
  m        12000?                     1000  0.5*24000; modem, V.32bis (TCM/Trellis); fax, V.17 (fax, TCM/Trellis modulation)
    c      12500                   80  960  NICAN
  m        14400               8            1.5*9600; modem, V.32bis (TCM/Trellis), V.33 (4-wire), HST (USRobotics); fax, V.17 (TCM/Trellis modulation); mobile GSM CSD (2G); CP2101
    c      15625                   64  768  NICAN
    c      16000                  62.5 750  NICAN; CP2102
  m        16800                            modem, US Robotics HST version (unusual)
! mS  i    19200   0x000e/EXTA 6       625  modem, V.32terbo/V.32ter (TCM/Trellis, non-ITU-T, AT&;T, unusual), V.34; some faster multimeters, MODBUS, Profibus, LIN; CDPD, Cellular Digital Packet Data; EXTA for external baudrate generator A
    c      20000                   50  600  NICAN
  m        24000                       500  modem, V.34
    c      25000                   40  480  NICAN
  ms       28800               4            3*9600; modem, V.34 (3200 baud, TCM/Trellis), V.FC/V.FastClass/V.FAST (non-ITU-T, Hayes/Rockwell); CP2101
  m        31200                            modem, V.34+/V.34bis
!  Sc      31250                   32  384  serial MIDI, NICAN; Profibus PA (trapezoidal biphase Manchester encoding)
           32000                       375
    c      33300?                           single-wire CAN
  m        33600                            modem, V.34+/V.34bis (3429 baud, TCM/Trellis), upstream for V.90/V.92 when digital off; near Shannon limit for 3kHz bandwidth
!  s       38400   0x000f/EXTB 3      312.5 38400 baudrate often needs a specific kludge; EXTB for external baudrate generator B
    c      40000                   25  300  NICAN
  m        40800                            some wideband modems
     i     45450                   22  264  Profibus
  m        48000                       250  2*24000 modem, upstream for V.92
  m c      50000                   20  240  NICAN, some wideband modems
           51200                            unusual; CP2102
  m        53300                            modem, V.44
  m        56000                            56 kbit/s line; modem, V.90 (8000/3429 baud, 56k/33k6, digital), V.92 (8000/8000 baud, 56k/48k, digital), various "56K"; frame relay; CP2101
! mS       57600   0x1001      2            1.5*38400, 3*19200; common higher speed lower than 115200; mobile HSCSD, GPRS (2.5G)
    c      62500                   16  192  2*31250; NICAN
  m        64000                      187.5 ISDN single, DS0/E0/T0; frame relay; CP2102
           72000                      500/3 3*24000
           74880                            ESP8266 post-reset/bootloader (26 MHz crystal) (115200*26/40)
           75000                  40/3 160  something industrial
     i     76800             2/3            2*38400; BACnet (Building Automation and Control) MS/TP networks; CP2102
    c      80000                  12.5 150  NICAN
     i     93750                  32/3 128  3*31250; Profibus
    c     100000                       120  NICAN, TIRA-1 infrared transceiver
!  S      115200   0x1002      1            3*38400, 2*57600; common factory default (8N1) for higher speeds; IrDA SIR; very common bootloader and general communication rate for higher speed devices (arduino...); typical max speed of older RS232
    C     125000                    8   96  4*31250; MIDI, NICAN, CAN (Fault-tolerant CAN ISO 11898-3, truck-trailer CAN ISO 11992)
  m c     128000                            CAN, ISDN dual; frame relay; CP2101
    c     153600             4/3            4*38400; CAN; CP2102; maximum of 16550 UART?
    c     160000                        75  NICAN
     i    187500                        64  Profibus
          192000                            frame relay (3*64000)
    c     200000                    5   60  NICAN
!  S      230400   0x1003    1/2            2*115200; AppleTalk/LocalTalk (RS485); highest "usual" UART bitrate; top for FTDI at 12 MHz?
!  SC     250000                    4   48  802.15.4; common higher-rate speed (eg. Marlin CNC), DMX512, MIDI, CAN (ISOBUS (ISO 11783), SAE J1939), NICAN; CP2102
    c     256000                            CAN; frame relay; CP2102
          320000                            frame relay
          384000                            frame relay
    c     400000                        30  NICAN
          448000                            frame relay
   s      460800   0x1004    1/4            4*115200
!  sC     500000   0x1005           2   24  CAN, NICAN, Profibus, max of Fieldbus; CP2102
          512000                            frame relay
          576000   0x1006    1/5            5*115200; IrDA MIR slow; CP2102
          691200             1/6            6*115200
   sC     800000                        15  NICAN
  m       921600   0x1007    1/8            8x115200; default speed of higher speed powerline modems
!  sC    1000000   0x1008           1   12  CAN 2.0 maximum (Highspeed CAN, ISO 11898-2, SAE J2284), NICAN, common RS485
         1024000                            frame relay
         1152000   0x1009   1/10            10x115200; IrDA MIR fast
         1382400            1/12
     i   1500000   0x100a                8  Profibus
         1536000                            USB 1.0 low-speed; frame relay
         1612800            1/14
         1843200            1/16            original 8250 UART clock; ESP8266?
!    i   2000000   0x100b         1/2    6  common RS485; top for FTDI at 48 MHz?
         2048000                            frame relay
         2073600            1/18
         2304000            1/20            20*115200
         2500000   0x100c         2/5
         2686400                            ESP8266?
         2764800            1/24
         2768000?                           serial UART maximum?
         3000000   0x100d         1/3    4  ceiling of some USB-serial chips with 12 or 48 MHz clock; Profibus
         3072000
         3500000   0x100e
         3686400            1/32
         3916800            1/34
         4000000   0x100f         1/4    3  IrDA FIR
         4608000            1/40            40*115200, ESP8266 top UART speed?
     i   6000000                  1/6    2  Profibus
     i  10000000                 1/10       RS485 maximum at 12m
    Ci  12000000                 1/12    1  Profibus maximum; CAN FD max (ISO 11898-2 2015)
    C   15000000                            CAN FD maximum

ioctl calls and structures

The serial port state is described in ioctl structure, termios. The structures are as follows:

asm-generic/termios.h                    asm-generic/termbits.h
#define NCC 8                  #define NCCS 19         #define NCCS 19
struct termio { struct termios { struct termios2 { unsigned short c_iflag; tcflag_t c_iflag; tcflag_t c_iflag; /* input mode flags */ unsigned short c_oflag; tcflag_t c_oflag; tcflag_t c_oflag; /* output mode flags */ unsigned short c_cflag; tcflag_t c_cflag; tcflag_t c_cflag; /* control mode flags */ unsigned short c_lflag; tcflag_t c_lflag; tcflag_t c_lflag; /* local mode flags */ unsigned char c_line; cc_t c_line; cc_t c_line; /* line discipline */ unsigned char c_cc[NCC]; cc_t c_cc[NCCS]; cc_t c_cc[NCCS]; /* control characters */ }; }; speed_t c_ispeed; /* input speed */ speed_t c_ospeed; /* output speed */ };

The termios structure is almost identical to termio, with exception of more indirect specification of data type and more control characters (19 instead of 8).

The termios2 structure is almost identical to termios, with added fields for integer-specified input and output data rate. (Few port controllers allow different speeds in each direction, this is uncommon. Usually both input and output data rate match.)

The c_cc structure is an array of control characters; position in array corresponds to a function, value corresponds to character

The bit flags for control lines are as follows:

bit    TIOCM_[ref]
0x0001  LE        in     DSR line    Line Enable
0x0002  DTR       OUT    DTR line
0x0004  RTS       OUT    RTS line
0x0008  ST                           secondary transmit, usually unused
0x0010  SR                           secondary receive, usually unused
0x0020  CTS       in     CTS line
0x0040  CAR,CD    in     DCD line
0x0080  RNG,RI    in     RI line
0x0100  DSR       in     DSR line
0x2000  OUT1      OUT?               Unassigned Programmable Output 1, usually unused
0x4000  OUT2      OUT?               Unassigned Programmable Output 2, usually unused
0x8000  LOOP      out    loopback    usually unused or used only for testing

ioctls as follows:

CLOCAL:

termios data structure flags:[ref]

c_cflag, control mode flags - speed, parity, bits

0x100f CBAUD           MASK: baud rates
 0x0000         B0        no speed - hang up
 0x0001        B50
 0x0002        B75
 0x0003       B110
 0x0004       B134
 0x0005       B150
 0x0006       B200
 0x0007       B300
 0x0008       B600
 0x0009      B1200
 0x000a      B1800
 0x000b      B2400
 0x000c      B4800
 0x000d      B9600
 0x000e     B19200,EXTA   on some systems, external-clock input A
 0x000f     B38400,EXTB   on some systems, external-clock input B; 38400 baudrate often needs a specific kludge
0x1000 CBAUDEX         mask for extended baudrates flag, for rates not defined in POSIX.1
 0x1000     BOTHER        other unspecified baud rate
 0x1001     B57600
 0x1002    B115200
 0x1003    B230400        highest usual value; top for FTDI at 12 MHz?
 0x1004    B460800
 0x1005    B500000
 0x1006    B576000
 0x1007    B921600
 0x1008   B1000000
 0x1009   B1152000
 0x100a   B1500000
 0x100b   B2000000        top for FTDI at 48 MHz?
 0x100c   B2500000
 0x100d   B3000000
 0x100e   B3500000
 0x100f   B4000000
0x0030 CSIZE           MASK: data bits
 0x0000        CS5        5 bits
 0x0010        CS6        6 bits
 0x0020        CS7        7 bits
 0x0030        CS8        8 bits
0x0040  CSTOPB         stop bits - 1 when unset, 2 when set
0x0080  CREAD          enable receiver
0x0100  PARENB         parity enabled
0x0200  PARODD         odd parity - even when unset, odd when set
0x0400  HUPCL          lower modem control lines (hang up) after last process closes device
0x0800  CLOCAL         ignore modem control lines

c_iflag, input mode flags - input data handling/translations

0x0001  IGNBRK      ignore BREAK if set; if IGNBRK=BRKINT=0, BREAK is received as 0x00 byte, if PARMRK set BREAK received as 0xFF 0x00 0x00
0x0002  BRKINT      if IGNBRK not set, BREAK flushes I/O queue and (if allowed) sends SIGINT to terminal
0x0004  IGNPAR      ignore framing and parity errors
0x0008  PARMRK      meaningful if IGNPAR=0,INPCK=1; if set, parity error byte 0xXX is received as 0xFF 0x00 0xXX
0x0010  INPCK       enable input parity check
0x0020  ISTRIP      strip off 8th bit, MSB
0x0040  INLCR       translate 0x0A to 0x0D (LF to CR)
0x0080  IGNCR       ignore 0x0D
0x0100  ICRNL       translate 0x0A to 0x0D (CR to LF, if IGNCR not set)
0x0200  IUCLC       translate uppercase to lowercase
0x0400  IXON        output XON/XOFF flow control enable
0x0800  IXANY       if set, any character restarts output (not just XON/START/^Q)
0x1000  IXOFF       input XON/XOFF flow control enable
0x2000  IMAXBEL     non-POSIX; ring bell if queue full; linux ignores, acts like always set
0x4000  IUTF8       non-POSIX; in linux, allows correct erasing of characters with utf8

c_oflag, output mode flags - output data handling/translations

0x0001  OPOST       implementation-defined input processing
0x0002  OLCUC       translate lowercase to uppercase
0x0004  ONLCR       translate 0x0A to 0x0D (LF to CR)
0x0008  OCRNL       translate 0x0D to 0x0A (CR to LF)
0x0010  ONOCR       no output of 0x0D/CR on column 0
0x0020  ONLRET      don't output 0x0D/CR
0x0040  OFILL       send fill characters for delay (keep line busy instead of pausing)
0x0080  OFDEL       not in linux; if set, fill char is 0x7F/DEL, otherwise fill char is 0x00
0x0100  NLDLY       MASK: newline delay (for mechanical terminals)
 0x0000    NL0
 0x0100    NL1
0x0600  CRDLY       MASK: carriage return delay (for mechanical terminals)
 0x0000    CR0
 0x0200    CR1
 0x0400    CR2
 0x0600    CR3
0x1800  TABDLY      MASK: horizontal tab delay (for mechanical terminals)
 0x0000    TAB0
 0x0800    TAB1
 0x1000    TAB2
 0x1800    TAB3,XTABS   expand tabs to spaces
0x2000  BSDLY       MASK: backspace delay (for mechanical terminals, not implemented)
 0x0000    BS0
 0x2000    BS1
0x4000  VTDLY       MASK: vertical tab delay
 0x0000    VT0
 0x4000    VT1
0x8000  FFDLY       MASK: form feed delay
 0x0000    FF0
 0x8000    FF1

c_lflag, line discipline flags - terminal control, control characters processing

0x0001  ISIG        if set: if INTR, QUIT, SUSP, DSUSP received, generate corresponding signal
0x0002  ICANON      canonical mode
0x0004  XCASE       if set if ICANON set, terminal uppercase-only (not in linux)
0x0008  ECHO        echo input to output
0x0010  ECHOE       if ICANON set, if set ERASE erases preceding input character, WERASE erases word
0x0020  ECHOK       if ICANON set, if set KILL erases current line
0x0040  ECHONL      if ICANON set, echo 0x0A/NL even if ECHO not set
0x0080  NOFLSH      disable flushing input/output queues when generating signals for INT, QUIT, SUSP
0x0100  TOSTOP      send SIGTTOU
0x0200  ECHOCTL     (non-POSIX) if ECHO set, show control chars other than 0x09/TAB, 0x0A/NL, START, STOP as ^x (eg 0x08/DEL = ^H)
0x0400  ECHOPRT     (non-POSIX) if ECHO set if ICANON set characters printed as they are erased
0x0800  ECHOKE      (non-POSIX) if ICANON set, if set, KILL erases each character on line
0x1000  FLUSHO      (non-POSIX, not in linux) output being flushed, triggered by typing DISCARD char
0x2000  PENDIN      (non-POSIX, not in linux) all chars in input queue reprinted when char read
0x4000  IEXTEN      implementation-defined input processing
0x8000  EXTPROC

Canonical mode - sends data line by line (on CR or LF); noncanonical mode - sends each byte immediately

Raw mode - noncanonical, no echo, no special processing:

c_cc[] array, control characters

pos      char     hex  ^x  POSIX Linux needed-flags
 0  ETX  VINTR     03  ^C              ISIG          sends SIGINT
 1  FS   VQUIT     1c  ^\              ISIG          sends SIGQUIT
 2  DEL  VERASE    7f                  ICANON        also 0x08, BS, ^H
 3  NAK  VKILL     15  ^U              ICANON        also ^X; erases input since last EOF
 4  EOT  VEOF      04  ^D              ICANON        send line immediately without witing for EOL; if first char of the line, send EOF/end of file
 5       VTIME      0                  not-ICANON    time in milliseconds for noncanonical read
 6       VMIN       0                  not-ICANON    minimum number of characters for noncanonical read
 7       VSWTCH     0        no   no   n/a           SystemV shell switch
 8  DC1  VSTART    11  ^Q              IXON          XON, start output after suspended by VSTOP
 9  DC3  VSTOP     13  ^S              IXON          XOFF, stop output until VSTART typed
10  SUB  VSUSP     1a  ^Z              ISIG          send SIGTSTP signal
11       VEOL       0                  ICANON        end of line
12  DC2  VREPRINT  12  ^R    no        ICANON,IEXTEN reprint unread characters
13  SI   VDISCARD  0f  ^O    no   no          IEXTEN (start/stop discarding pending input)
14  ETB  VWERASE   17  ^W    no        ICANON,IEXTEN word erase
15  SYN  VLNEXT    16  ^V    no               IEXTEN quotes next input, removes special meaning
16       VEOL2      0        no   no   ICANON        (another end of line)
    EM   VDSUSP   (19) ^Y    no   no   ISIG,  IEXTEN (sends SIGTSTP)
    DC4  VSTATUS  (14) ^T    no   no   ISIG?         (display status, send SIGINFO - not in linux)

default value (from strace):
   0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18
 x03 x1c x7f x15 x04 x00 x00 x00 x11 x13 x1a x00 x12 x0f x17 x16 x00 x00 x00

VTIME and VMIN control read() behavior:
VTIME  VMIN
0       0           read() immediately returns, with zero length (no data) if no byte available
0       nonzero     read() blocks until VMIN bytes available
nonzero 0           read() returns after VTIME, with or without data
nonzero nonzero     read() returns after VTIME or after VMIN bytes available

terminal-initiated signals (when ISIG enabled)


Serial-over-TCP/IP, RFC-2217

The age of modems is mostly over, the stage was seized by computer networks. Ethernet, wifi, and many other technologies replaced serial lines. Except in communication with various devices - the serial, usually in the form of TTL-level UART (Arduino, ESP8266, various microcontrollers) or RS232 (various peripherals) or RS485 (industrial buses), is still a king there. Even many USB devices act as serial ports, whether as /dev/ttyACM* or /dev/ttyUSB*.

The problem was encountered even in the old times. The telnet protocol, short for "teletype network", was born. And extended a number of times, to add features as the era required.

Telnet itself is good for "plain" sending of data streams back and forth, with some additional negotiation functions. However, serial ports have a number of "out of band" features, controlled by ioctls - everything from bitrate to modem control lines to sending a break.

But there are ways for encoding such out-of-band activity as in-band signaling.

RFC 854, other telnet RFCs

RFC854

For telnet itself, the signaling is based on a dedicated escape character, 0xFF, named IAC, "interpret as command". After it, one or two other bytes are sent with the command and a parameter.

(if 0xFF has to be send through as a character, send it twice; first time it gets interpreted as IAC, the next command 0xff then means "send 0xff through".)

definition:

Several codes are defined:

Options: [ref]

RFC 2217

The 0x2C option for the telnet serial port operation includes subcommands:

TODO: examples of command sequences

Control sequences, telnet console examples

The responses to commands do not have to arrive in order.

CAUTION: The commands are not particularly synchronized with the data output (due to data buffering between the running program and the port output), so eg. commands to handle control lines can be executed before the nearly preceding data bytes made their way out of the port to freedom (or enslavement in another hardware).

For clarity, sequences "ff fa" and "ff f0" are written together as "fffa" and "fff0".

ff fb 2c IAC WILL COM-PORT
ff fb 01  IAC WILL ECHO
ff fe 01  IAC DONT ECHO
ff fb 03  IAC WILL SUPPRESS-GO-AHEAD
ff fd 01  IAC DO ECHO
ff fd 03  IAC DO SUPPRESS-GO-AHEAD
ff fb 2c IAC WILL COM-PORT
ff fd 2c IAC DO COM-PORT

identify terminal type:

ff fd 18  IAC DO TERMINAL-TYPE
ff fb 18  IAC WILL TERMINAL-TYPE
fffa 18 ?? fff0  IAC SB TERMINAL-TYPE SEND IAC SE
fffa 18 00 ..... fff0  IAC SB TERMINAL-TYPE IS <code> IAC SE

stop a process in terminal

[press ctrl-c on keyboard]
ff f4
[remote side sends SIGINT to running process]
(duplicate of direct-sending of ctrl-C, same principle different mechanism as not all ancient terminals could do this)

Control sequences, serial port examples

...picocom initialization on connection to virtual port
ff fb 2c IAC WILL COM-PORT // local-computer is able to act as a serial port
ff fb 01  IAC WILL ECHO // remote-port is able to send echo
ff fe 01  IAC DONT ECHO // local computer does not want remote to echo
ff fb 03  IAC WILL SUPPRESS-GOAHEAD // remote port is able to suppress the half-duplex GOAHEAD control
ff fd 03  IAC DO SUPPRESS-GOAGEAD // local computer wants remote to not send GOAHEAD
ff fd 00  IAC DO BINARY // remote port wants local to send binary data
ff fb 2c IAC WILL COM-PORT // remote port is able to act as a COM port
ff fd 2c IAC DO COM-PORT // local computer wants remote to act as a COM port
ff fd 2c IAC DO COM-PORT // remote port wants local computer to act as a COM port
fffa 2c 01 00 00 25 80  fff0  IAC SB COM-PORT SET-BAUDRATE 0x00002580  IAC SE // set remote baud rate to 9600, aka 0x2580
fffa 2c 6b 00  fff0  IAC SB COM-PORT MODEMSTATE-MASK 0x00  IAC SE // remote port wants to not send any modemstate updates
fffa 2c 02 08  fff0  IAC SB COM-PORT SET-DATASIZE 0x08  IAC SE // set remote data size to 8 bits
fffa 2c 03 01  fff0  IAC SB COM-PORT SET-PARITY 0x01  IAC SE // set remote parity to none
fffa 2c 04 01  fff0  IAC SB COM-PORT SET-STOPSIZE 0x01  IAC SE // set remote stopbits to 1
fffa 2c 05 01  fff0  IAC SB COM-PORT SET-CONTROL 0x01  IAC SE // execute remote command 0x01, "set flow control to none"
fffa 2c 6b 00  fff0  IAC SB COM-PORT rNOTIFY-MODEMST 0x00  IAC SE // remote confirms modemstate mask
fffa 2c 65 00 00 25 80  fff0  IAC SB COM-PORT rSET-BAUDRATE 0x00002580  IAC SE // remote confirms baudrate set to 9600
fffa 2c 66 08  fff0  IAC SB COM-PORT rSET-DATASIZE 0x08  IAC SE // remote confirms data size set to 8 bits
fffa 2c 67 01  fff0  IAC SB COM-PORT rSET-PARITY 0x01  IAC SE // remote confirms parity set to none
fffa 2c 68 01  fff0  IAC SB COM-PORT rSET-STOPSIZE 0x01  IAC SE // remote confirms stopbits set to 1
fffa 2c 69 01  fff0  IAC SB COM-PORT rSET-CONTROL 0x01  IAC SE // remote confirms command to flow control none
...set RTS+ (send SET-CONTROL 0x0b == "Set RTS Signal State ON")
fffa 2c 05 0b fff0  IAC SB COM-PORT SET-CONTROL 0x0b IAC SE // execute remote command 0x0b, "Set RTS Signal State ON"
fffa 2c 69 0b fff0  IAC SB COM-PORT rSET-CONTROL 0x0b IAC SE // remote confirms command to set RTS to ON
...set RTS- (send SET-CONTROL 0x0c == "Set RTS Signal State OFF")
fffa 2c 05 0c fff0  IAC SB COM-PORT SET-CONTROL 0x0c IAC SE // execute remote command 0x0b, "Set RTS Signal State OFF"
fffa 2c 69 0c fff0  IAC SB COM-PORT rSET-CONTROL 0x0c IAC SE // remote confirms command to set RTS to OFF

Serial port over network

Sometimes it is needed to make a remote serial port appear as local one, typically with TCP/IP network in between. Telnet-based serial connection can be utilized.

The same control sequences defined in original telnet are extended with the 0x2C (44d) code for the serial port related operations.

Here we need two ends of communication:

linux servers

rfc2217_server.py - good

https://github.com/pyserial/pyserial/blob/master/examples/rfc2217_server.py

Python has a highly powerful library for handling serial ports, aptly named pyserial. rfc2217_server.py is a reference implementation of such a server.

Very handy for debugging, can be easily edited to serve other purposes.

Fairly complete support of pretty much all the baudrate and control line commands.

Runs on any platform on which Python 3 can be installed.

 python3 ./rfc2217_server.py -v -v -v -v -v -p 5000 /dev/ttyUSB0

(runs daemon on port 5000, physical port /dev/ttyUSB0, verbose about what it is doing)

ser2net - good?

Written in C. Available as a standard package for many distros (incl. Raspbian, OpenWRT). Runs as a daemon.

TODO: find out how good command support it has, if it can handle arbitrary baudrates.

 ser2net

(configuration in /etc/ser2net.conf)

remserial - bad (no RFC2217)

http://lpccomp.bc.ca/remserial/ Does not seem to correctly handle the RFC2217 commands. Good enough for basic shoveling data back and forth without need to handle control lines or change speeds.

Fairly simple C implementation but no support for RFC2217 commands.

linux client

ttynvt - good

https://gitlab.com/lars-thrane-as/ttynvt

Written in C. Userspace-implemented character device driver, via CUSE. Capable of handling ioctls native-like.

Runs in a master process and two slave threads that handle the communication itself. When using strace to debug what's happening, don't forget to attach it to both the master process PID and two other (usually subsequent) PIDs or a lot of activity will appear to be missing.

Sends debug information to syslog. Copious data can be obtained by setting syslog level properly.

 ttynvt -D 9 -M 199 -m 6 -n ttyNVT0 -S 192.168.1.10:5000

(runs daemon that creates /dev/ttyNVT0 with major:minor devnode 199:6, and on port-open connects to 192.168.1.10:5000 where it expects a RFC2217-compliant server)

remserial - bad (no ioctls)

http://lpccomp.bc.ca/remserial/

Written in C. Available for many distros. Can create local devicefile-like interface, linked to a pseudoterminal (pty).

No support for ioctl interception.

socat - bad (no ioctls?)

Written in C. Available for many distros. Can create local devicefile-like interface, linked to a pseudoterminal (pty). Does not seem to be able to intercept ioctls.

TODO: check

socat-rfc2217 (todo: check)

http://www.dest-unreach.org/socat/contrib/socat-rfc2217.html

windows client

hercules terminal

Not a device driver, only a userspace terminal for handling local serial ports and some connections to remote devices. A versatile tool for simplest terminal operations, supports hex.

todo: look at its proprietary extensions for GPIOs, implement the calls to ttyvnt

HW VSP 3

https://www.hw-group.com/software/hw-vsp3-virtual-serial-port

Free Virtual Serial Ports

https://freevirtualserialports.com/


ttynvt modifications

The original implementation of serial port did not support arbitrary baudrates, uses the termios structure. Newer variant is not limited by the hardcoded baudrate list, uses extended termios2 structure and modified ioctl calls.

ttynvt of course did not support these. (Because nothing can be perfect. Oh well, let's start with good-enough.)

To obtain support for arbitrary baud rates, and for compatibility with newer software and it's ioctl calls, the missing ioctls were crudely hacked into the original ttynvt implementation.

possible future expansion

Experiences were obtained with handling ioctl calls, allowing to further extend the virtual port functionality - add "vendor-specific" command sequences, eg. for handling various GPIO and bitbanging modes on USB-serial dongles.

Server-side then also has to be modified for support of such.

Virtual port servers can be implemented with relative ease, to appear like a RFC2217-compliant remote serial port (and to appear like a real local serial port in the system). Emulation of various devices (eg. position-faking GPS, for testing or "testing" purposes) is one of the options. Or maybe a protocol translator between one vendor's software and another vendor's device. (This can also be done locally, using the same CUSE trick ttynvt does to act as a proxy between a local serial port, but a python implementation of just the server side may be easier to both write and debug.)

The extensions for control signals can be used for including such out of band data over RS485. A two-port microcontroller to translate messages, split data stream from command stream, handle packetization/addressing for the bus.

issues

esptool.py

esptool.py was crashing on port open:

esptool.py --port /dev/ttyNVT0 chip_id
esptool.py v2.7-dev
Serial port /dev/ttyNVT1
Connecting...
Traceback (most recent call last):
File "/usr/miniconda3/lib/python3.7/site-packages/serial/serialposix.py", line 501, in read
'device reports readiness to read but returned no data '
serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected or multiple access on port?)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/esptool.py", line 2966, in <module>
_main()
File "/usr/local/bin/esptool.py", line 2959, in _main
main()
File "/usr/local/bin/esptool.py", line 2653, in main
esp = ESPLoader.detect_chip(each_port, initial_baud, args.before, args.trace)
File "/usr/local/bin/esptool.py", line 261, in detect_chip
detect_port.connect(connect_mode)
File "/usr/local/bin/esptool.py", line 462, in connect
last_error = self._connect_attempt(mode=mode, esp32r0_delay=False)
File "/usr/local/bin/esptool.py", line 442, in _connect_attempt
self.sync()
File "/usr/local/bin/esptool.py", line 381, in sync
timeout=SYNC_TIMEOUT)
File "/usr/local/bin/esptool.py", line 334, in command
p = self.read()
File "/usr/local/bin/esptool.py", line 279, in read
return next(self._slip_reader)
File "/usr/local/bin/esptool.py", line 1875, in slip_reader
read_bytes = port.read(1 if waiting == 0 else waiting)
File "/usr/miniconda3/lib/python3.7/site-packages/serial/serialposix.py", line 509, in read
raise SerialException('read failed: {}'.format(e))
serial.serialutil.SerialException: read failed: device reports readiness to read but returned no data (device disconnected or multiple access on port?)

crude solution:

in slip_reader(): replace

        read_bytes = port.read(1 if waiting == 0 else waiting)
with
        try:
          read_bytes = port.read(1 if waiting == 0 else waiting)
        except (OSError, serial.serialutil.SerialException):
          print("[caught error: no data]")
          continue

download


If you have any comments or questions about the topic, please let me know here:
Your name:
Your email:
Spambait
Leave this empty!
Only spambots enter stuff here.
Feedback: