Because often it's handy to control oscilloscopes, and other instrumentation, over the network. Because wireless connection is often more handy than a network cable. Because cheap Raspberry Pi Zero W can be used to retrofit a USB-only device for full-scale wifi operation.
The implementation is specific for Rigol DS1054Z (and Rigol DS1052D, todo).
The DS1054Z scope has Ethernet and USB. The USB port is a standard instrument driver USB-TMC class interface. The Ethernet port is a standard LXI interface.
The scope gets a "backpack" with a Raspberry Pi computer. The raspi is connected to the scope by a short USB cable, accessing the TMC interface.
The raspi may or may not have a display of its own and a touchscreen or set of buttons, for more operations (sending screenshots and acquired data to the server, server-based protocol decoding, voice annotations, store banks of configurations, probe gains for nonstandard or ganged-together probes...).
 When reverse-engineering mixed signal systems, it's often needed to see both the DC analog level and the weaker AC signal riding on it. Switching between DC and AC coupling and adjusting the gain for each signal is a pain in the weknowwhere. Adapter that couples together two channels, and then separate setting of one channel as DC and the other as more sensitive AC is a good hack, but it halves the input impedance (typ. a megaohm) and screws up the probe attenuation and the voltages are incorrect. One-touch attenuation setting to the not-in-the-common-list probe value can be handy. (Also it may be possible to tweak the two-to-one adapter with a trimpot to set up the apparent gain.)
The raspi needs to run a daemon for the network connectivity, acting as a gateway between a TCP/IP socket and the TMC device. A python solution was chosen, for flexibility.
Adding functionality to the python daemon will be easy - HTTP-based screenshots and waveform requests, touchscreen interface, iGornet-class MQTT data feeds...
The "LXI" client, with corresponding "liblxi" library, was chosen.
The measurement instrumentation is a wild mess of standards old and new, of abstraction layers good and poor.
The standards can be split to the hardware and the software/data halves; the SCPI commands can be relayed through GPIB, USB-TMC, RS232/485, LXI, whatever.
Essentially, the hardware/interface/communication standards are about setting up a way to send SCPI messages to the instruments and getting responses; in some variants also synchronizing time between instruments and sending them triggers.
VISA - "Virtual instrument software architecture", abstraction layer over different communication standards
high-speed buses, between instrument cards, within one compact unit - motherboard or backplane and cards
over TCP/IP, usually ethernet-based, can be wireless (then beware of unpredictable delays and increased roundtrips)
The VXI-11 protocol is essentially the GPIB interface transferred to Ethernet link.
The functionality is emulated by three network connections:
The low-level control messages are:
The session setup is fairly involved, beginning with the RPC portmapper call. The SCPI messages are wrapped in the GPIB-derived protocol (very similar to USBTMC).
For whatever reasons, many services do not run on fixed-assigned ports and rely on dynamically assigned ports via a portmapper service. The most popular one is SunRPC, also called ONC RPC. The portmapper calls request the port of the service via port 111, then connect to where they were told.
The services requested are identified by a number. Some are:
Many non-ancient instruments feature a USB-B port (sometimes microUSB), with USB TMC class device.
In Linux, after attachment the device enumerates and forms a /dev/usbtmc* device file. This can be further interfaced with as a character device.
The device behaves similar to usual tty terminals/serial ports with a few important differences. (Eg. usual COM-over-IP attempts won't work.)
The device is usually used to communicate the SCPI command strings and their responses. Eg.
A detour to the ancient era of GPIB, when the features were born.
data bit 0 DIO1 1 ] [ 13 DIO5 data bit 4 data bit 1 DIO2 2 ] [ 14 DIO6 data bit 5 data bit 2 DIO3 3 ] [ 15 DIO7 data bit 6 data bit 3 DIO4 4 ] [ 16 DIO8 data bit 7 End-or-identify EOI 5 ] [ 17 REN Remote Enable Data valid DAV 6 ] [ 18 GND/DAV Not ready for data NRFD 7 ] [ 19 GND/NRFD No data accepted NDAC 8 ] [ 20 GND/NDAC Interface clear IFC 9 ] [ 21 GND/IFC Service request SRQ 10 ] [ 22 GND/SRQ Attention ATN 11 ] [ 23 GND/ATN shield 12 ] [ 24 SG signal/logic ground
Uniline commands - single control line involved
Multiline commands - 2 or more control lines involved
Addressed messages - "multicast", addressed by bit flags to one or more devices at once
STB, Status Byte:
ESR, Event Status Register byte:
ESE, Event Status Enable
USB488 is a sub-class of the USBTMC class, implementing various GPIB/IEEE488 features.[ref]
The device is half-duplex. New communication to the instrument must not be sent while there is still a response undelivered.
Each message is prepended by a header, for message synchronization and protection against data loss; a GPIB spec.
12-byte message, same size as bulk header; output-to-device only
MsgID - 1 byte - MsgID=128/0x80 for Trigger cmd - 3 bytes 0x00 - 8 bytes - reserved
a 12-byte header prepended to the USBTMC messages, both to and from the device
MsgID - 1 byte - message identifier bTag - 1 byte - varies with each transfer bTagInv - 1 byte - bTag, inverted bits, a form of checksum 0x00 - 1 byte - reserved size - 4 bytes, LSB first - size of the message flags - 1 byte - bmTransferAttributes, command-specific flags 0x00 - 3 bytes - reserved ...then the message itself, terminated by \n (0x0a)
...then 0x00 padding to multiple-of-4 length
End of message specified by termination with 0x0a character, and also with EOM bit in flags
USB MsgID codes:
USBTMC_MSGID_DEV_DEP_MSG_OUT = 1 USBTMC_MSGID_REQUEST_DEV_DEP_MSG_IN = 2 USBTMC_MSGID_DEV_DEP_MSG_IN = 2 USBTMC_MSGID_VENDOR_SPECIFIC_OUT = 126 USBTMC_MSGID_REQUEST_VENDOR_SPECIFIC_IN = 127 USBTMC_MSGID_VENDOR_SPECIFIC_IN = 127 USB488_MSGID_TRIGGER = 128
result status codes:
USBTMC_STATUS_SUCCESS = 0x01 USBTMC_STATUS_PENDING = 0x02 USBTMC_STATUS_FAILED = 0x80 USBTMC_STATUS_TRANSFER_NOT_IN_PROGRESS = 0x81 USBTMC_STATUS_SPLIT_NOT_IN_PROGRESS = 0x82 USBTMC_STATUS_SPLIT_IN_PROGRESS = 0x83 USB488_STATUS_INTERRUPT_IN_BUSY = 0x20
USBTMC_REQUEST_INITIATE_ABORT_BULK_OUT = 1 USBTMC_REQUEST_CHECK_ABORT_BULK_OUT_STATUS = 2 USBTMC_REQUEST_INITIATE_ABORT_BULK_IN = 3 USBTMC_REQUEST_CHECK_ABORT_BULK_IN_STATUS = 4 USBTMC_REQUEST_INITIATE_CLEAR = 5 USBTMC_REQUEST_CHECK_CLEAR_STATUS = 6 USBTMC_REQUEST_GET_CAPABILITIES = 7 USBTMC_REQUEST_INDICATOR_PULSE = 64
USB488_READ_STATUS_BYTE = 128 USB488_REN_CONTROL = 160 USB488_GOTO_LOCAL = 161 USB488_LOCAL_LOCKOUT = 162
Sometimes the device gets to a weird state after a read, and a subsequent read timeouts (timeout in usb/backend/libusb1.py, from self.bulk_in_ep.read in usbtmc/usbtmc.py). This usually occurs after the device was closed and opened again. Rerunning the command then works again.
Many devices have quirks; Rigol scopes are common with subtle problems or protocol weirdness. Workarounds are often implemented in drivers and libraries, which is complicated by later firmwares correcting the issue and then not working with the workaround (eg. Rigol DS1054Z with 00.04.04.SP4 firmware, which needs self.rigol_quirk=False set in usbtmc/usbtmc.py despite setting it True from the device type autodetection; to add a worm to the can, the firmware revision is not available in the USB data available from the port).
Standard Commands for Programmable Instruments - general specification for text-based communication protocol, based on IEEE 488.2 (1987 and 1992 flavors), a command set of IEEE-488 aka HP-IB aka GPIB
Simple one-line commands, hierarchical structure, ":" as hierarchy delimiter, "*" for non-hierarchical command (eg. *IDN?, "identify yourself"). Some devices remember the last "directory" in the hierarchy, others (Rigol DS1054Z, I am looking at YOU!) don't and always require full absolute "paths". The absolute path starts with ":".
SCPI commands have to tell the instrument they are terminated. Over stream links (TCP socket, serial port, direct character device read/write...) the delimiter is \n aka 0x0a. In packet connections (USBTMC, UDP...) the message itself can act as its delimiter. Some instruments however require the \n termination even then. It is safer to use it even if not needed.
Other possible line termination characters/delimiters encountered are \r aka 0x0d, and \0 aka NUL aka 0x00.
In VISA, the delimiters are set by read_termination and write_termination properties of the instrument object.[ref]
The SCPI statements consist of one or more "words", separated by spaces.
The first word can be a query (ends with "?", response from the instrument is expected) or a command (data are written to the device, operation is performed...).
The statements have optional parameters. For example:
The statements are case-insensitive.
The statements can have a long and short form, often written in mixed case together; DISPlay can be used as both DISP and DISPLAY (but usually not as DISPL), both upper and lower and mixed case.
Command sets, even for the same general thing, can vary widely, even between devices from the same vendor.
Eg. for sending a screenshot: (from lxi-tools plugins)
Or for acquire memory depth, where DS1052D/E takes :ACQ:MEMD norm/long, and DS1054Z takes :ACQ:MDEP <number>.
Chaos inherent for many-actors system. The only worse situation would be if it'd be designed by a committee.
DS1052 screenshot, more modern firmware (?)
DS1052 waveform buffer data
DS1052 screenshot, more modern firmware (?)
DS1054Z waveform buffer data, long form; 600,000 bytes read
CAUTION: some commands work only in RUN state, others only in STOP state!
Usually the SCPI command response is a plaintext string. Some types of data (screenshots, raw waveforms...) require a binary blob.
The responses are usually plaintext strings, with numbers as decadic, with exponential notation (eg. 4.546875e-02), and are terminated with newline (\n, 0x0a).
When the response starts with #, it has a different format:
The Arbitrary Block Data is composed of a header and a variable length binary blob of payload:
Presence of this header at the initial response can be used for continuous read from the source (device, port, socket...) until the payload is all received.
With direct device read from usbtmc/usb488, further binary bytes are present. These are usually stripped by the reading plugin.
The #-something binary data header can be used even with shorter binary data, eg. direct transfer of 32 or 64 bit float numbers where the ASCII representation could lead to unwanted loss of precision.[ref]
The responses, including the binary ones, are always followed by newline character (\n).
Some older devices (DS1052, I am looking at YOU!) may not conform and may send the data without the header. The read routines then do not know when to stop and have to either rely on the known data lengths (possible false positives) or on the read timeout (introduces delay).
The VISA/VXI-11 architecture specifies strings for device addressing. The string uses double colon :: as delimiter and is composed of:
The common interfaces are:[ref]
Other ones, less common with cheapo instruments, are:
For the server (usbtmc-server.py), python was chosen for flexibility and ease of coding.
For the client (liblxi/lxi-tools), C was chosen for the speed of execution (no overhead with loading megabytes of python libraries).
for local Rigol DS1054Z scope, exposing SCPI-raw on default port 5025:
python3 ./usbtmc-server.py --backend python_usbtmc USB::0x1ab1::0x04ce::INSTR
the same for Rigol DS1052D (no long-reads):
python3 ./usbtmc-server.py --backend python_usbtmc USB::0x1ab1::0x0588::INSTR
python3 ./usbtmc-server.py --backend linux_kernel /dev/usbtmc0
Uses several different backends:
The linux_kernel backend had to be modified for the long-data read for the screenshots and data.
On the first read, the beginning of the response is checked if it starts with #<digit>; if not, do the read as before. If yes, loop through reads until the indicated payload length is read.
The python server can get a thread for listening on HTTP, for commands like screenshots, waveforms, data previews, save/restore settings, anything else.
Another thread can read buttons or touchscreen.
The scope then will have exposed LXI interface (SCPI-raw), screenshots over HTTP, iGorNet control over MQTT, local buttons/screen control, arbitrary protocol decoders...
Python interface for USBTMC devices, also used by universal_usbtmc.
Handles various device quirks.
In usbtmc/usbtmc.py (eg. /usr/local/lib/python3.7/dist-packages/usbtmc/usbtmc.py), there are detections of quirks of various devices, set as flags for workarounds in the code. Eg.
The self.rigol_quirk was set for the DS1054Z scope, however the 00.04.04.SP4 firmware does not need them. The setting has to be disabled.
It is possible to modify the library to apply the quirks only for certain serial numbers of devices. Then the software will behave even for the same devices with different firmwares.
A little more involved code can enable/disable the quirk flags based on the *IDN? response (TODO?).
The lxi command connects to the remote either through VXI-11 (SunRPC over port 111, default) or directly (SCPI-raw over port 5025, option -r; in original lxi the option applies only to SCPI commands, not to screenshots).
get identification from device "wifiscope" (in /etc/hosts), using raw interface
lxi scpi -r -a wifiscope '*idn?'
get screenshot from device "wifiscope" using rigol-1000z plugin, using raw interface (not in stock lxi)
lxi screenshot -r -p rigol-1000z -a raspidispw screenshot.png
get waveform data (not in stock lxi) from device "wifiscope" using rigol-1000z plugin, using raw interface (not in stock lxi)
lxi getdata -r -p rigol-1000z -a raspidispw wav.bin
The liblxi was left intact.
The lxi command of lxi-tools got patched to check the beginning of the response if it starts with #<digit>; if not, do the read as before. If yes, it is the Arbitrary Block Data; loop through reads until the indicated payload length is read.
The screenshot plugin for Rigol 1000Z was amended with this read patch as well.
When the ABD header is not present, the data can be still retrieved using the -L option.
For scpi, the command parsing was modified to allow up to three space-separated arguments. No more need for quotes for simple commands.
Additional options for DS1052D/E workarounds:
The communication of the options to the screenshot plugin is somewhat crude; it is done by making the options.h options structure accessible from the plugin.
Patches again lxi-1.21
The master sends a UDP broadcast request to port 111, with a fixed discovery payload.
static char rpc_GETPORT_msg = 0x00, 0x00, 0x00, 0x02, 0x00, 0x00,[0x00, 0x03,] 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, [0x00, 0x06, 0x07, 0xaf,]0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00 };0x0003 is the GET_PORT RPC call, 0x000607af is the VXI-core channel
Present slaves respond with another UDP packet from their port 111. Master then initiates the "standard" RPC-mediated VXI-11 "*IDN?" request.
PORT STATE SERVICE 80/tcp open http 111/tcp open rpcbind 111/udp open rpcbind 617/tcp open sco-dtmgr // RPC-assigned VXI-11 channel, abort 618/tcp open dei-icda // RPC-assigned VXI-11 channel, core 619/tcp open compaq-evm // RPC-assigned VXI-11 channel, interrupt 5353/udp open|filtered zeroconf // mDNS, but doesn't seem to work 5555/tcp open freeciv // Rigol-flavor scpi-raw 6000/udp open|filtered X11
Picture Transfer Protocol
The USB interface can be selected to either USBTMC or PTP mode.
Let's try ptpcam from libusb:[ref]...
ERROR: Could not get device info!
The ptp_getdeviceinfo call, a PTP_OC_GetDeviceInfo ptp request, does not get through.
The screenshots can be already acquired via LXI or USBTMC, so this is a minor concern.
Apparently the PictBridge thing acts as a client that connects to a server (usually a printer) and sends data to print. The server-side software, which establishes connection with the PictBridge device (a camera, or here the scope), was too difficult to locate.
The scope becomes visible on USB connection in "Devices and Printers" under "Unspecified" section as "DS1000Z Series". (Driver installation can take a couple minutes.)
In Device Manager, the scope is seen under "Portable devices" as "MTP USB device". Generic WpdMtp driver is apparently used.
In TMC mode, it shows in Device Manager as "USB Test and Measurement Device (IVI)".
Instrument Model: DS1104Z Manufacturer: RIGOL TECHNOLOGIES Serial Number: DS1ZA210400549 Description: LXI Class: LXI Core 2011 LXI Version: 1.4 Host Name: MAC Address: 00-19-AF-xx-xx-xx IP Address: 10.0.0.116 Firmware Revision: 00.04.04.SP4 VISA TCP/IP String: TCPIP::10.0.0.116::INSTR Auto-MDIX Capable: NO VISA USB Connect String: USB0::0x1AB1::0x4CE::DS1ZA210400549::INSTR
Network Hardware Configuration Status: CONFIGURED Password: Not Specified Link Speed And Duplex Negotiation: Automatic Link Speed: 100 Mbps Duplex: Full Automatic
direct connection to the network, using ethernet cable and socket on the scope
faster response time than USBTMC/wifi, but lower bulk transfer speed
LXI benchmark: 100 ID requests, 154.3 requests/second
action bytes response total ident 56 0.001 0.02..0.05 screenshot/bmp 1152054 1.67..1.78 1.88..2.00 screenshot/png ~88000 0.05..0.08 0.77..0.81 live data read 1200 0.02..0.05 0.04..0.07 mem read 8192 0.005 0.06..0.11 mem read 16384 0.008 0.07..0.12 mem read 125000 0.34..0.51 0.39..0.65 mem read 250000 0.32..0.67 0.67..0.80 max length of read at once, need to partition to pieces mem read calculated 524288 ~0.9 ~1.4 mem read calculated 1048576 ~1.8 ~2.8 mem read calculated 12M ~20.2 ~32.2 mem read calculated 24M ~40.3 ~64.3
connected over USBTMC to raspi3, with wifi link over python server as above
LXI benchmark: 100 ID requests, 57.5 requests/second
action bytes response total ident 56 0.02..0.03 0.04..0.05 screenshot/bmp 1152054 0.97..1.16 1.50..1.64 screenshot/png ~32000 0.53..0.55 0.57..0.60 live data read 1200 0.02..0.05 0.04..0.07 mem read 8192 0.03..0.04 0.05..0.06 mem read 16384 0.03..0.04 0.06..0.08 mem read 125000 0.22..0.24 0.30..0.34 mem read 250000 0.43..0.44 0.55..0.60 max length of read at once, need to partition to pieces mem read calculated 524288 ~0.9 ~1.1 mem read calculated 1048576 ~1.8 ~2.2 mem read calculated 12M ~20.2 ~25.0 mem read calculated 24M ~40.3 ~49.9
allowed values (without the commas):
single channel AUTO 12,000 120,000 1,200,000 12,000,000 24,000,000 two channels AUTO 6,000 60,000 600,000 6,000,000 12,000,000 four channels AUTO 3,000 30,000 300,000 3,000,000 6,000,000
Do the change in :RUN mode or the scope will ignore the command. (todo: check)
Compared to 1054, 1052 is much older and weaker. The most basic SCPI functionality is present; a lot of controls can be done.
The screenshots aren't normally working. The :DISP:DATA? command is not present.
It is however possible to get the raw framebuffer data as a constant-size 320x234 ".raw" image, with 8 bits per pixel, with colors packed in RR-GGG-BBB scheme (MSB to LSB). Use :HARDCOPY and then :LCD:DATA?.
The firmware revision 00.02.02.02.00 of course does not start the bulk transfers with the #number-length preamble so read until timeout is needed. Dislike.
The long read from memory seems to be failing with python_usbtmc backend (and works with modified linux_kernel backend).
:ACQ:MEMD LONG/NORMAL works only after next run/stop cycle; :WAV:DATA? done after acq:memd give the length when the memory was acquired.
The scope also has a RS232 port. Its speed is between 9600 and 38400 bps (selectable in menu). While a bit too slow (estimated 20 seconds for a screenshot, 4.5 seconds for 16k memory, and over 4.5 minutes for the megabyte of samples), it is still good enough for using the scope with an arduino. An "emergency" wireless connection could be made with ESP8266 and serial-to-tcp RFC2177 interface.
Caution: the long memory reads take quite some time, up to 25 seconds for the entire megabyte. This may appear as a timeout to the unwary.
Rough timing of reads, response time (between issuing command and starting receiving data) and total command running time (against raspi running usbtmc-server.py, over wifi with ping time between 5..12 msec):
LXI benchmark: 100 ID requests, ~22.5 requests/second
action bytes response total screenshot 74880 1.74-1.77 1.81-1.96 (raw, without compression) live data read 600 0.03-0.04 0.05-0.06 mem read, 2ch 8192 0.19-0.21 0.21-0.23 mem read, 1ch 16384 0.41-0.43 0.43-0.46 longmem read, 2ch 524288 11.9-12.3 12.1-12.5 longmem read, 1ch 1048576 24.4-24.8 24.9-25.9
:hardcopy (is it necessary?)
:LCD:DATA? (returns 74880 bytes of raw LCD image, 320x234, 8bpp RGB 2:3:3)
Patched LXI can do the conversion to PNG on demand.
TODO: server-side (usbtmc-server.py) emulation of :DISP:DATA? with PNG conversion
NORM LONGsingle channel 16384 1048576 (0.2/12.5 seconds read) two channels 8192 524288 (0.4/25.5 seconds read)
The :acq:memd setting reflects to the amount of data read by :wav:data? only after the scope was run, triggered, and stopped.
Other useful commands:
The LXI/SCPI infrastructure allows very simple interfacing of instrumentation.
E.g. a simple python server can listen on messages from MQTT with data, cache the last one, and serve it on a request. (Or look up the last value from a log. Or whatever.) A meteo station then can be implemented as a ESP8266 sensor with periodic MQTT feed, and the measurement can be made available via eg. ":MEAS:OUTTEMP?" query from standard LabView/MatLab/anything with VISA libraries.
Such "virtual instruments" ("simstruments"?) can be also made visible on the LAN easily via mDNS.
(do not confuse with Picture Transfer Protocol)
PTP, IEEE 1588, is a tool for time synchronization between devices on a LAN, similar to but more accurate than NTP and alternative to GPS sync (not needing the receiver at each node, not needing GPS signal reception).
In linux, it is available as the linuxptp package, with ptp4l command. Uses timemaster to run PTP with NTP as reference clocks (PTP for precision sync of relative time, NTP for less accurate absolute time).
GPS is commonly used to synchronize time of machines. The receivers provide absolute time usually in NMEA-formatted strings over a serial line, and precision-edge pulse-per-second sync as a separate digital signal. (Take good care about the line impedance and other transmission line factors when handling this signal; if the edge has to have properly short rise time, the line must not attenuate even very high frequencies.)
On unix/linux computers, the signal is usually connected to the DCD input of the serial port, as by RFC2783 and https://www.kernel.org/doc/Documentation/pps/pps.txt.
This PPS sync can be leveraged even without GPS (or the GPS can be simulated completely including the time-bearing NMEA sentences, using a shared single-Tx-to-many-Rx bus). A master clock can send the PPS (with or without the NMEA), the slaves then take the data as if it was a real GPS receiver.
USB-serial is polling-based. The interface is queried many times per second. This introduces a possibly significant jitter (up to 125 microseconds on USB2, much more on USB1).
The same could be possibly implemented locally via microcontroller-and-nRF24L01 combination. The packets can be delivered one-to-many, the delays are the same (if retransmissions are disabled, which in one-to-many they have to be anyway as the delivery confirmation would only throw in chaos), missing a couple packets doesn't throw the local clock off significantly, and it's cheap.
Any other packet radio with low and constant latency will do the job too. Raw pulse-per-second can be sent via a very simple radio too, with caveats for the rf noise.
For distributed dataloggers, log also some metadata with the timing quality - absent or spurious sync pulses may then help explain discrepancies in stored data, allow to reliably know the timestamps are unreliable.
Optical methods are also possible, whether free-space or fiber-guided.
For very high precision, a good shared wire can do the job. Same principle as for the PPS syncing, but missing pulses can be highly detrimental for the equipment-group operation.
For pulse-based or command-based triggering over wireless, be wary of jitters and especially of missing packets. Wireless, especially 2.4 GHz in urban areas, can get clogged fairly randomly - usually just the millisecond before DAQ for an unrepeatable test has to be started.
For lower precision, shared-sync clock can do, with a clock offset to trigger at. The offsets then can be unicasted or repeatedly broadcasted to each equipment piece in advance, with assured delivery of at least one command to each device.