This manual is intended for telescope operators and computer programmers. It explains how to start, stop, and maintain the Telescope Control Computer (TCC) control system.
Log into the appropriate tcc as user "tcc":
Set up the desired version in eups:
lsst
setup tcc
setup tcc -t test
setup tcc <version>
eups list tcc
To start the tcc: start up its actors, then start the tcc:
For the 3.5m telescope: sec start tert start tcc start For the SDSS 2.5m telescope: prim start sec start tcc start
For the 3.5m telescope:
sec start tert start tcc start
For the SDSS 2.5m telescope:
prim start sec start tcc start
To restart the tcc or start a different version of the tcc: stop the tcc, restart the other actors, then start the tcc again:
For the 3.5m telescope: tcc stop sec restart tert restart tcc start For the SDSS 2.5m telescope: tcc stop prim restart sec restart tcc start
tcc stop sec restart tert restart tcc start
tcc stop prim restart sec restart tcc start
To stop the TCC: stop the tcc, then stop the other actors:
For the 3.5m telescope: tcc stop sec stop tert stop For the SDSS 2.5m telescope: tcc stop prim stop sec stop
tcc stop sec stop tert stop
tcc stop prim stop sec stop
You can stop or restart actors as needed, in any order, but:
hub startNubs tcc
tcc mirror connect
The full list of start, stop and restart commands for each actor is:
stop
If needed, you can communication with the tcc and mirror controllers using telnet. More than one user can be connected at the same time (e.g. you and the tcc or hub). Telnet to the appropriate machine using the appropriate port:
For example: telnet tcc35m 3532 to talk to the 3.5m secondary mirror controller.
telnet tcc35m 3532
Warning: do not talk to the axis controllers while the tcc is tracking or slewing. You will confuse the tcc and halt the telescope.
3.5m telescope logs are on hub35m:
SDSS 2.5m telescope logs are on sdss-hub
In case of an emergency: PUSH A RED EMERGENCY STOP BUTTON.
Please only do this in an emergency, because it is hard on the enclosure. If safety permits and you have a connection to the TCC try issuing an Axis Stop command first.
Axis Stop
The TCC supports running batch jobs in the background via the Queue command. Batch jobs are python files in $TCC_DATA_DIR/jobs. Each batch job is one twistedActor.ScriptRunner script. These are very similar to TUI scripts, with some important differences:
Queue
With this in mind, please see $TCC_DATA_DIR/jobs/example.py for an example, and read TUI's Scripting Manual for more information.
$TCC_DATA_DIR/jobs/example.py
Warning: TCC batch jobs are run in the same event loop as the TCC itself. You can easily kill tracking by having your script use too many resources. If your script requires lengthy computations, run it as a thread using yield sr.waitThread. If your script needs to pause use yield sr.waitSec. If batch jobs do prove to interfere with tracking, it would be fairly straightfoward to modify the Queue command to run batch jobs as separate processes.
yield sr.waitThread
yield sr.waitSec
The hardware controllers are small computers that talk directly to the axis motors and encoders, mirror actuators and so on. The TCC communicates with them in plain text via TCP/IP sockets.
Exactly what initialization means to a controller depends on the controllers. However, in general initializing will stop any motion (if applicable) and generally put the controller into a known sensible state. Initialization is a normal thing to do to a controller. Initialization is done via the commands specific to each controller. For example Axis Init and Mirror Init. If initialization fails, you may have to restart the hardware controller.
Axis Init
Mirror Init
To send a brief command to an axis or mirror controller, use the Talk command.
Talk
For a more extensive session with a mirror controller, disable the collimation process in the TCC with Proc Disable Collimate (to avoid confusing the TCC) and telnet to the mirror controller port. The mirror controllers support multiple users, so you need not worry about the TCC's connection.
Proc Disable Collimate
For a more extensive session with an axis controller, stop the axes in the TCC, then use talk2boxes. (Eventually we expect that direct telnet will work, but it will not work while we are stuck using the terminal servers.)
The directory $TCC_DATA_DIR data directory, which points to ~/tccdata, contains the following configuration files:
The data directories for the 3.5m and SDSS 2.5m tccs are git repositories hosted here: 3.5m and SDSS 2.5m. Using git makes it easy to see how data files have changed over time. Here are some useful git commands for keeping the data up to date:
git status
git add filepath
git commit -a
To view changes I suggest running git gui in directory $TCC_DATA_DIR on the TCC. This gives a nice graphical view. (If you have a clone of a git repository on a Mac, then it is even nicer to view changes using the free program SourceTree).
git gui
$TCC_DATA_DIR
The TCC software requires earth orientation predictions to operate at full accuracy. The data is read from $TCC_DATA_DIR/earthpred.dat and this this section describes how to keep that file up to date. Predictions are regularly e-mailed to us by the U. S. Naval Observatory, as a file called "IERS Bulletin A". Data from this bulletin is extracted by a utility program, and written out in a form the TCC software can use.
$TCC_DATA_DIR/earthpred.dat
Obtain a copy of IERS Bulletin A. Normally this is electronically mailed to the site about once a week. If that fails you can download the bulletin from the US Navel Observatory. As of 2014-05 the web page for bulletins is http://www.iers.org/IERS/EN/Publications/Bulletins/bulletins.html and the link for the latest IERS Bulletin A is http://datacenter.iers.org/eop/-/somos/5Rgv/latest/6. If you save the file to $TCC_DATA_DIR/iersbulla.dat then earthFilt.py will find it automatically.
Convert the data to a format suitable for the TCC:
$ earthFilt.py
If the TCC software is running you may load the new data using the TCC command SET TIME. Otherwise it will be loaded automatically at startup.
IERS Bulletin A contains predicted values for the rotation of the earth, as UTC - UT1, and pole wander (used for a small correction to longitude and latitude). It also gives the exact value for TAI-UTC at the start of the predictions. IERS Bulletin A is set up for the convenience of people who keep UTC, and so want UTC-UT1. But the TCC uses TAI (and TAI-UT1); TAI is smoothly varying, whereas UTC has occasional abrupt one-second changes, called "leap-seconds". Thus the TCC has a conversion utility that derives TAI-UT1 (and TAI-UTC) from the bulletin's UTC-UT1 predictions and the initial value of TAI-UTC. It deduces the presence of leap-seconds by noting discontinuities in the value of UTC-UT1.
The TCC software reads Earth orientation data from $TCC_DATA_DIR/earthpred.dat. The TCC extracts the predictions for yesterday, today, and tomorrow and saves these predictions to the Earth global data block. You may update the predictions file at any time. The new file will be read when the TCC next synchronizes its clock to the time reference, and you may force this event (though that is rarely necessary) by issuing the TCC command SET TIME.
The current values of TAI-UT1 and pole position are obtained by reading the Earth block and performing the appropriate interpolation. There is a TCC subroutine which does all this.
Information about the position and orientation of the instrument and guide camera is contained in instrument data files, contained in the $TCC_DATA_DIR/inst/ directory. This information is necessary for proper tracking, guiding and pointing data collection. Some of the information is a matter of taste or convention, e.g. the desired center of an instrument, but much of it must be carefully measured for the telescope to operate correctly.
The instrument data is distributed among four kinds of files. These are discussed in File Types. The names of the files determine the names used in the SET INSTRUMENT command, as explained in File Names . For help measuring instrument parameters, see the Instrument Data Fitting manual.
Instrument data files are only read during a SET INSTRUMENT command. Hence to change the values the TCC is actually using, you must not only edit the file(s), but also load the new data by issuing the appropriate SET INSTRUMENT command.
There are four kinds of instrument-related data files: default, instrument position, guide camera view, and instrument. Each instrument/guide-camera-view combination has at most one of each kind of file associated with it. The files are read in the order listed, with data from later files overriding data from earlier files.
The format is identical for all of these files. This provides maximum flexibility, but the heavy overlap also means you must be very careful about which entries you specify and which you leave unchanged (by omitting them or commenting them out). The files are loaded in the order listed above, and in the event of a conflict (two files setting a value differently), the last data seen is used, without warning. Note that the default file and instrument file are always read, but the instrument position file is only read if it exists, and the view file is only read if a non-default view is specified.
The name of each instrument and view (as used by the SET INSTRUMENT command) is specified by the name of the corresponding instrument-specific or view data file. Hence the naming of instrument data files is very important. The files are named as follows:
DEFAULT.DAT
IP_
pos
.DAT
V_
pos_view
I_
pos_inst
where:
inst
SET INSTRUMENT=
view
SET INSTRUMENT/VIEW=
All files are kept in the directory $TCC_DATA_DIR/inst/
The "angle from the instrument x axis to the rotator x axis" is degenerate with the rotator's "fixed physical position" (except there may be a sign change). I recommend setting the rotator's "fixed physical position" to zero and measuring "angle from the instrument x axis to the rotator x axis" as described below.
This is a common configuration for measuring telescope pointing. It is especially common right now, when we don't have a working offset guider. In such a case, some data fields are zero, and other fields are clones of existing fields; if you ever change one field, you must remember to change its clone!
Position of the guide camera in the guide image frame = position of the center of the rotator in the instrument frame. If you change one, remember to change the other!
Angle from the guide image x axis to the guide camera x axis = angle from instrument x axis to rotator x axis. If you change one, remember to change the other!
There is no guide camera positioner. All position and orientation data for it should be set to zero.
InstEdit is a simple utility which allows you to interactively vary certain elements of the current Inst global data block. This may help you try out or play with values for an instrument data file. Once you have determined the correct values, you must still manually edit them into the appropriate instrument data file.
To run InstEdit, exit from the TCC (or better yet, open a second connection to the TCC, one for TCC commands, one for InstEdit) and type: $ InstEdit. For more information, see the TCC Utilities Manual.
$ InstEdit
Don't forget to edit your changes into the appropriate instrument data files when you are through, else all changes will be lost.
The basic procedure is as follows (but see the note for the SDSS primary).
tcc proc disable collimate
telnet tcc25m 2532
tcc proc enable collimate
While homing, watch the mirrors. If anything goes wrong, abort homing using mirror controller command: stop
You can abort the relax job (or any job) using tcc queue stop
tcc queue stop
While homing you should see the mirrors do the following:
This section describes how to move mirrors for collimation and other engineering purposes. Before moving a mirror, it is important that you understand how it is mounted. There are some quirks and gotchas that may not make sense otherwise.
The basic procedure is:
tcc proc disable coll
tcc talk sec "move 987, -23.5, 65.2, 57.6, -3.4"
tcc talk sec "offset 112, 34, -12.3"
Before moving a mirror it is very important to understand how it is physically supported. Otherwise you may induce stresses in the mirror, deforming the surface (especially a problem for the SDSS Primary) or the resulting motion may not be at all what you expect.
All on-axis mirrors (all mirrors except the 3.5m tertiary) use the following coordinate axes. With the telescope at the horizon, and with you standing behind the primary mirror looking through the hole in the primary at the secondary:
Hence:
Hardware-specific limitations and unusual features of the mirrors are as follows:
There are three axial actuators and no transverse actuators. Transverse position is maintained by a set of fixed links, so you can only command piston, x tilt and y tilt. (With the old TCC, when commanding tertiary mirror piston it was necessary to also command an equal amount of y translation. That is no longer the case.)
The coordinate system for the 3.5m Tertiary is unusual because it is desired that x/y tilts move the image sideways or up and down, rather than in the funny arcs that would result using the coordinate system for on-axis mirrors. Thus:
Thus:
To put it another way, with the telescope at zenith, if you are standing at an instrument port looking in at the tertiary then:
The SDSS primary mirror is supported by four sets of air-actuated belloframs. Three axial actuators and one transverse "vertical" (y translation) actuator adjust the hard points for the servo system that drives the belloframs. Two transverse "lateral" links adjust x translation; one is mounted near the top of the mirror (at horizon) and the other at the bottom. The lateral links act through air-actuated force fuses which release automatically whenever the lateral links move, as otherwise the lateral actuators would stall. The transverse belloframs have bearings to allow free motion, but the axial belloframs have a fairly high friction coupling to the back of the mirror.
The primary mirror cannot simply be translated at will; the friction is too great in the axial belloframs. The mirror may not move the full distance and in any case will have residual stresses that deform the surface figure. Piston and reasonably small tilts should be fine. To move the mirror in translation one should first move the actuators and then relax the support system as follows:
Queue Run=Relax
The TCC software uses a "telescope model" to correct for various repeatable hardware errors, such as flexure of the secondary mirror support, and non-perpendicularity of the azimuth and altitude axes. The model contains a term for each identified hardware error. The coefficients for the model are determined empirically, by measuring pointing error, and fitting the result to the model using tpoint.
The model is contained in the TelMod global data block. TelMod is set at startup from file $TCC_DATA_DIR/telmod.dat, and remains fixed unless somebody explicitly changes it (as described below under "Load...").
The procedure for taking pointing error data and fitting a model is as follows:
Set Inst=
Telescope: Pointing Data
tccdata/ptdata/
To analyze your pointing data:
cd tccdata/ptdata
tpoint
inmod ../telmod.dat
indat pointing_data_file_name
Then fit the model. See the tpoint manual or tpoint help for more information. You may add or delete terms, though please be very cautious about doing so.
help
To update the TCC's pointing model:
outmod ../telmod.dat
telnet localhost 3500
set block telmod/in=telmod
When you are done taking pointing data and possibly modifying the pointing model, please save the results to the tccdata git repository:
git add *.dat
git push
git push fails
git pull
The TCC has an extensive set of motion limits to protect the telescope. These include position, velocity, acceleration, and jerk limits for the azimuth and altitude axes, instrument rotator, and guide camera positioner. They also include position (only) limits for the primary, secondary and tertiary mirrors and guide camera. The TCC's limits only apply to motion requests by the TCC. The axes controllers have their own limits, which they will not exceed.
Position limits behave in a straightforward manner. If a move is attempted which violates position limits of the TCC or an axis controller, the motion is rejected with an appropriate error message. Under normal circumstances only the TCC should ever complain; an axis controller's limits will only come into play if the TCC's limits are made too broad (a serious configuration error) or the user bypasses the TCC and talks directly to the axis controller.
Regarding velocity and acceleration, the TCC must leave some headroom between the TCC's and controller's limits. With insufficient headroom, the axes controllers will be late finishing slews. This is a serious problem, as the TCC pre-computes slew paths, and relies on the axes controllers following the paths accurately and arriving on time. Note that the axes controllers do not complain if velocity or acceleration are exceeded during a move. They simply reduce the speed or acceleration as required and continue trying to keep up as best they can.
Jerk (the rate of change of acceleration) is only limited by the TCC, and then only when computing a slew. The limit may be set to practically anything. Its main purpose is to protect people from abrupt motions; it seems a bit unlikely that the motors can introduce enough jerk to cause problems for the equipment.
All motion limits are stored in the AxeLim block. Most limits are fixed; these are read from the file $TCC_DATA_DIR/AXELIM.DAT when the TCC software starts up and then left alone. Some limits depend on which instrument is selected, and these are set in the same fashion as all other instrument-specific data.
To change fixed (instrument-independent) limits: Edit the file $TCC_DATA_DIR/axelim.dat. Load the changes into the TCC software by issuing the TCC command: set block axelim/in=axelim Set the current instrument again (to update instrument-specific limits).
set block axelim/in=axelim
To change instrument-specific limits see Specifying Instrument Data.
The Tune block contains values used to "tune" the operation of the TCC software. These values can usually be left alone, but may require adjustment if the hardware changes, or if something is not working correctly. The various tuning parameters are described below (grouped into several general categories).
To change tuning parameters, edit the file $TCC_DATA_DIR/tune.dat. For your changes to take effect, you must then load the file by executing the TCC command: set block tune/in=tune
$TCC_DATA_DIR/tune.dat
set block tune/in=tune
Note: you must configure the TCC computer's clock to keep TAI (not UTC). This requires an NTP server that keeps TAI (which can fairly easily be set up if you have a GPS clock). An alternative is to rewrite the function tcc::tai to read UTC and convert it to TAI.
source $LSST_HOME/loadLSST.sh
eups distrib install <packagename>
which python
pip install twisted
git@bitbucket.org:csayres/pytcc.git
scons
sudo scons install
pip install RO
svn+ssh://sdss3svn@sdss3.org/repo/ops/PACKAGE_NAME/trunk
git@github.com:r-owen/twistedActor.git
eups declare -r . name version --current
opscore
git
trunk
git@github.com:r-owen/coordConv.git
eups declare -r . coordConv version --current
scons install declare version=version
eups setup -r .
cd /usr/local/bin
makeTCCStartupScript -h
makeTCCStartupScript telescope >tcc
makeMirrorCtrlStartupScript -h
makeMirrorCtrlStartupScript mirror >shortName
chmod +x tcc shortName1 shortName2
For example:
cd /usr/local/bin makeTCCStartupScript 35m >tcc makeMirrorCtrlStartupScript sec35m >sec makeMirrorCtrlStartupScript tert35m >tert cd /usr/local/bin chmod +x tcc sec tert
TCC_DATA_DIR
axelim.dat
earthpred.dat
telmod.dat
tune.dat
jobs
Individual software packages are installed and managed using git and eups (scons commands use eups under the hood). Type "lsst" at the command line to setup the eups environment. Note it is expected that all packages have been initially eups declared (see Installing Software) by the time you get here! Also note that git and eups are separate systems, so it is up to the human typing to keep versions/tags coherent between the two.
1) When code is ready for a new tag (beta or otherwise) use git:
2) Update code on TCC machine.
Declare a new version for a package. pathToPackage should be a natual location, usually in a common directory with previous versions of the package. A good strategy is to copy the complete package from the working git or svn repo to pathToPackage before declaring it.
This will make the new version visible in eups list, and available for use with setup.
Make this version of the package current.
Undeclare a packageVersion, wiping eups' memory of it, the code is left on disk.
Equivalent to undeclare the version and delete the code from disk.
Remove all built software and unit test results
Build all software and run unit tests. If unit tests are mysteriously failing try using the -j 1 flag to force all tests to be run on a single processor. This seems to be required occasionally on tcc35m. Many of the tests are necessarily asyncronous, and running on a single processor seems to help. Unit test and corresponding logs are stored in tests/.tests, you should look in there after building software. Try "ls tests/.tests/*failed" to identify failed tests after building.
Use this command from within any scons-managed package to declare a new version from the current state of the current directory. If current is specified with the command, this new package version will be made current. Code is automatically copied to a logical location before it is declared.
The Message Keyword manuals for the tcc and mirror controllers are generated from the keyword dictionaries in the opscore package, as follows:
Check the status of the TCC with Show Status.
Show Status
If the TCC appears to be doing what you asked of it and the axis controllers are happy, then check the position you specified with Show Object/Full. If the position or coordinate system looks wrong, check your track command very carefully. Did you specify the correct coordinate system?
Show Object/Full
Update the earth orientation predictions as per the procedure described here.
Clear the communications channel by initializing the controller. If the init times out, the odds are that the link or controller is down; check wiring, power, etc. If the init fails due to garbled communications then the terminal server or controller may be sick.
