Instrument Control Computer Design

Each instrument on the Apache Point Observatory 3.5m telescope has an instrument control computer (ICC) which is the public "face" of the instrument. The ICC receives commands, issue replies and writes data to disk. An ICC may also issue commands (via a separate connection to the hub).

This manual describes how to design an ICC for the APO 3.5m telescope.

Contents

• Introduction
• Interfaces
• Commands
  • Exposure Commands
• Data
• Replies
  • Reply Basics
  • Reply Format
  • Keyword Value Format
• Units

Introduction

The basic architecture at APO is as follows:

A central hub routes all communications. The hub receives commands from commanders (e.g. TUI users) and routes them to actors (e.g. instruments). Actors issue replies, which are sent to all commanders. The hub does not modify commands or replies, except for necessary changes to header information, plus correcting invalid formatting and possibly changing whitespace.

• Commanders include users connected via TUI (APO's graphical user interface), scripts in the hub, a few instruments that need to issue commands, and users connected via other means than TUI.
• Actors include instruments (via their ICC), the telescope (via the telescope control computer, or TCC) and scripts in the hub.

It is important to remember that the 3.5m telescope is a multi-user, remote controlled telescope. There are usually at least two users connected (scientist and telescope operator) and there may well be more (collaborators or scientists waiting their turn). In addition, even people working at the observatory are somewhat remote from the telescope. Thus:

• Your instrument should monitor itself as much as possible, so it can detect and report failures instead of mysteriously malfunctioning.
• Your instrument must protect itself. Incorrect commands (incorrectly sequenced, incorrectly formatted, you name it) should be handled gracefully. The ICC should accept those commands it can safely accept and reject the rest with a sensible error message. The ICC must do all it can to keep itself and its instrument operational.

Interfaces

The hub can talk to an ICC using TCI/IP or serial communications. The ICC should live in the computer room, if possible, since it is a warm environment and one capable of safely eliminating excess heat. However, if your ICC is sufficiently rugged and does not generate too much heat, and with approval from APO staff, your ICC may also live in the intermediate level of the telescope enclosure, or even at the instrument.

All communications between the control building and the telescope must be over optical fibers, to protect against lightning damage. We provide optically protected TCP/IP and some RS-232 serial ports. If you use some other protocol then you must provide the optical interfaces and possibly the fiber as well.

Commands

Commands are sent to the hub in the following format:
actor CmdrID command string

Where:

• actor is the actor for which the command is intended.
• CmdrID (CID) is a positive integer assigned by the actor to identify this command and its replies. (CmdrID must fit into a 32-bit unsigned integer variable. The number may eventually wrap around, but an actor should not reuse a number unless all replies have been received from the earlier command of the same number.)

The hub then strips the actor, assigns a new commander ID (because different users may use the same commander ID) and sends the otherwise-unmolested command to the actor as:
CmdrID MsgID command string
where MsgID is a historical relic used to make commands and replies process the same that you should ignore.

Each reply from the actor must be tagged with CmdrID (as per the reply format discussed below). The hub uses that number to figure out who sent the command and with what CmdrID, and it includes that information when it broadcasts the reply to all commanders.

The command string must consist of printable ASCII characters. Beyond that, the format is up to the designer of the ICC. However, we do have some suggestions:

• Commands should be readable. This makes looking through logs much simpler. Please do not over-abbreviate. Bandwidth is not a big issue and 99% of all commands will be machine-generated.
• Write a command parser. Please resist the urge to use a forth interpreter, tcl interpreter or shell as the command parser. This protects your instrument from accident or malicious hacking and allows you to issue better error messages for mal-formed commands.
• Protect your instrument. Design the instrument and command set such that any sequence of commands (valid or otherwise) either causes the expected behavior or is rejected with an explanatory error message. Thus, for example, you may very well receive a command while another command is executing. Be prepared to deal with it somehow.
• String data should follow basic C rules: enclose strings in double quotes and use \ to escape embedded " or \.

One ICC that makes a good model is the DIS ICC. Craig Loomis wrote the DIS ICC with the current hub in mind, and in Python, the preferred language.

Exposure Commands

The hub has a set of instExpose scripts that provide a unified interface for taking exposure. This offers advantages to both the ICCs and to TUI. The ICCs need only save exposures using the file name and location specified by the instExpose script (rather than trying to figure out where the exposure for this user belongs and what the next sequence number should be). And, of course, TUI takes advantage of the uniform exposure interface to use one code base to control exposures for all instruments.

All instruments should accept stop and/or abort commands while exposing, even if they do no other form of multitasking. Both commands halt the exposure, but stop reads out and saves the partial exposure, whereas abort discards it.

instExpose can support a wide variety of exposure commands. Nonetheless, if you are implementing a new command set from scratch, it will save trouble to emulate DIS or some other modern instrument.

Data

Instruments write data to an NFS-shared disk. The data should be written in FITS format with basic information in the FITS header, including bit depth and image shape. The hub then fills in the FITS header with WCS information, an exposure comment, etc., and notifies commanders that the image is available.

Replies

Each command results in one or more replies, ending with a reply saying that the command finished or failed. Actors may also output unsolicited replies, and this can be very useful if an instrument has environmental monitors. The instrument can report a problem right away using an unsolicited reply, rather than waiting until the hub sends a command.

A well chosen set of replies is the very heart of a well-designed ICC. Designing a command set is fairly easy because an instrument designer has a very keen sense of what the instrument should do. A good reply set is less obvious, yet is crucial for informing users as to what is going on inside the instrument. Thus this section starts with a section on overall design criteria before going into details of the reply format.

Reply Basics

Replies are the only way users know what an instrument is doing! When a user sends a command to your instrument, the other users basically don't know about it until they see the resulting replies. All users (even the one who sent the reply) rely on the instrument's replies to keep track of what is going on.

Thus an ICC should issue a reply whenever the instrument changes state. When an exposure starts, when readout starts, when filter change starts and finishes, when temperature is read, etc. Be chatty!

Here are some specific guidelines:

Slow commands

For commands that take a long time to execute, issue a reply when the command starts and another when it finishes (successfully or otherwise). Also:

• If you have a reasonable estimate of how long the command will take, output that estimate when the command starts. This allows TUI to present a countdown timer.
• If your instrument supports asking for status while a command executes, then your time estimate should consist of two values: total estimated duration of the command and the estimated time remaining. This allows TUI to correctly redisplay the countdown timer should a user ask for status while the command is executing.

Overall Health and Status

It is helpful to display a summary of overall health and status. One easy way to send this info to TUI is to send a "status word": a set of binary bits, each of which indicates one item of state. Suggested rules (which make the info easy to use):

• Every error, warning or other "busy" condition should have its own bit. Thus one could have bits for:
  • A motor is moving (busy)
  • A particular thermal sensor is out of range (warning)
  • A filter wheel is lost (error)
• All bits should be set to 1 for "something to report", 0 for "nothing to report".
• You may also wish to assign some bits to normal states such as "shutter open".
• Output the status word whenever it changes (of course!). Note: if your ICC cannot output unsolicited replies, then you may have have to delay the output; in that case, make sure the bit stays on until the next chance you get to output the status word. This assures that a transient problem will be seen.
• The status word is not meant to be a substitute for outputting important information via other, more readable, keywords. It is an simply an efficient way to generate an overall health and status summary. The resulting code in TUI is much more maintainable and much more likely to do the right thing than a spaghetti of if-then-elses based on various keywords.

Reply Format

Actors output replies in the following format:
CmdrID MsgID MsgType ReplyData

where:

• Messages are ASCII.
• Extra spaces are ignored.
• CmdrID (CID) is the ID number of the commander that sent the command that triggered this reply. This is only relevant if the actor allows more than one user to be connected at the same time. Such actors must assign a user ID number to each user and keep track of which user issued which command. The TCC is a good example of how a multi-user actor should handle user ID.
• MsgID (MID) is the ID number of the message that triggered this reply. Use 0 if the reply is unsolicited (i.e. not in response to any command).
• MsgType is a one-character message type code.
• ReplyData: message data in keyword-value format.

The hub then processes the header of the reply, resulting in the following being sent to commanders:
Prog.User CmdrID Actor MsgType ReplyData

where:

• Prog.User is the program name and user name of the user who sent the command that triggered the reply. For unsolicited replies, the program name is blank and the user name is the actor, e.g. ".tcc".
• CmdrID is the commander ID assigned by the user to the command that triggered the reply. 0 for unsolicited replies.
• Actor: the actor from which the reply came.
• MsgType is a one-character message type code.
• ReplyData: reply data in keyword-value format.

Technically speaking, the hub actually parses the message data and re-assembles it before sending the reply to the commanders. But the message data should not be affected unless it was mis-formatted.

Message Type Character

Each reply has an associated type character, as follows:

>	Command queued: the command was correctly parsed and has been queued for background execution.
i	Information: normal informational output; the commend will continue to execute.
w	Warning: something is unusual, but the command will continue to execute.
:	Finished: the command finished successfully.
f	Error: the command failed.
!	Fatal error: something is seriously wrong; the actor needs an init, reboot or power cycle.

Solicited replies (triggered by a command) must end with exactly one ":", "f", or "!" reply. Before that, you may issue as many "i" or "w" replies as you like. (You may also issue more than one ">" reply, but it violates the intent of that message code.)

Unsolicited replies (not triggered by a command, thus CmdrID = 0) do not have any such restrictions.

Keyword-Value Format

The data in a reply must be in keyword-value format. This consists of one or more keywords separated by semicolons; each keyword may also have one or more comma-separated values. For example:
NoValueKeyword; OneValueKeyword=val1; ManyValueKeyword=val1, val2, val3...

Keywords indicate the state of your ICC. Typically the user interface will set up a callback to be triggered by a particular keyword, so its display can be kept up-to-date. For this to work well, keywords should:

• Always be output when the relevant state changes. Remember that we can have more than one user at the same time. Other users should not have to parse commands to know what is going on. For example, if one user says "focus=50", you should output a keyword indicating the new focus.
• Self-contained and non-modal. The keyword data is much easier to use if all one has to know is the values for that one keyword, rather than looking at the rest of the reply, or worse, knowing what the ICC was doing at the time.

Examples:

• Use FiltDuration=sec instead of Duration=sec. Otherwise the user has to wonder "duration of what"?
• Use SecPos=pos1, pos2... instead of ID=Sec; Pos=pos1, pos2... Otherwise the user has to parse the ID keyword to understand what Pos applies to. If you have a lot of devices, you can include id as part of the data, e.g. Pos=id, pos1, pos2...

Details

The picky details about keyword-value format are as follows:

Basics:

• All characters must be ASCII
• Spaces are optional before and after = ; ,
• Extra spaces are ignored
• A zero-value keyword may not include a "="

Keywords are words. We suggest you stick to legal C identifiers, though the actual rules are a bit broader. Case is ignored.

Values can be any of:

• numbers (int or float or nan; ints can be in any base using C notation)
• dates and times (yyyy-mm-dd hh:mm:ss.sss or yyyy-mm-ddThh:mm:ss.s) please stick to UTC (or TAI if leap seconds are an issue); avoid local time if at all possible.
• Keywords (same rules as keywords)
• "-delimited strings (escape any contained " or \ with \), for example:
  "an \\ example \" string containing \\ and \""
  Note that the " delimiters are only required if the text is not a keyword (e.g. contains spaces or other special characters).

Units

Please use APO standard units for commands and replies. These include:

• Delta times in seconds (integer or float, not sexagesimal).
• Absolute times or dates may be sent in any of these forms:
  • hh:mm:ss.s...
  • yyyy-mm-ddThh:mm:ss.s... (ANSI standard format).
  • MJD, in decimal days (standard) or decimal seconds (less common).
• Wavelength in Angstroms (visible) or μm (IR).
• Small mechanical motions in μm (e.g. focus).
• Postion on the sky in degrees.
• Offsets on the sky in degrees or arc seconds.
• Vacuum pressure in torr.
• Temperature in C (visible) or K (IR).