The CEBAF accelerator at Thomas Jefferson National Accelerator Facility (Jefferson Lab) successfully began its experimental nuclear physics program in November of 1995 and has since surpassed predicted machine availability. Part of this success can be attributed to using the EPICS (Experimental Physics and Industrial Control System) control system toolkit. The CEBAF control system is one of the largest accelerator control system now operating. It controls approximately 338 SRF cavities, 2300 magnets, 500 beam position monitors and other accelerator devices, such as gun hardware and other beam monitoring devices. All told, the system must be able to access over 125,000 database records. The system has been well received by both operators and the hardware designers. The EPICS utilities have made the task of troubleshooting systems easier. The graphical and text-based creation tools have allowed operators to custom build control screens. In addition, the ability to integrate EPICS with other software packages, such as Tcl/Tk, has allowed physicists to quickly prototype high-level application programs, and to provide GUI front ends for command line driven tools. Specific examples of the control system applications are presented in the areas of energy and orbit control, cavity tuning and accelerator tune up diagnostics.

The 4 GeV CEBAF accelerator at Thomas Jefferson National Accelerator Facility (Jefferson Lab) is arranged in a five pass racetrack configuration, with two superconducting radio-frequency (SRF) linacs joined by independent magnetic 180^o transport arcs. The continuous electron beam is composed of three interlaced variable intensity beams that can be independently directed from any of the five passes to any of the three experimental halls. This allows three simultaneous experiments at the same or different energies and currents. Electrons are emitted through a thermionic cathode or a polarized laser cathode that is being commissioned. Presently only one experimental hall, hall C, is fully operational, with hall A in the final commissioning stages.

The control system for the accelerator is based upon EPICS (Experimental Physics and Industrial Control System) [1]. The control system follows the so-called standard model, which is a client server system consisting of a collection of UNIX workstations and X-terminals connected by a network to multiple servers running device control software [2]. CEBAF uses a switched Ethernet network which allows simple scaling to higher bandwidths as needed. The servers are VME based single board computers running the EPICS real-time database, and the client computers are HP work stations.

In 1993 CEBAF decide to convert from TACL (Thaumaturgic Automated Control Logic) to the EPICS control system [2]. The integration has been one of evolution where the two control systems were allowed to coexist as the change was being made. This made commissioning easier and did not overly stress the controls group.

Using the CEBAF control system as a platform, engineers, physicist and operators developed the tools necessary to commission and run the machine. Control screens have been built and customized as needed by both the operators and the hardware designers [3, 4]. High level applications programs for energy and orbit control have been developed through software packages such as tcl/tk [5, 6, 7, 8, 9]. The following discussion will describe in detail these applications of the CEBAF control system.

One of the problems with commissioning large accelerators is that many of the screens have been developed by the designers, engineers and physicists. The control screens they have built are rarely useful for the day-to-day operators. The expert screens contain too much information and detail to be useful. At CEBAF most of the screens have evolved into more user friendly screens designed by the operators. The EPICS graphical and text based creation tools, and scripts and other tools, are relatively easy to learn and operators have been encouraged to do this. Visual screens have been developed that allow the operator to identify problems quickly and make adjustments. These screens make it easy for operators to clear faults, initiate automated routines, access programs for documenting hardware problems and get to more complex expert screens for detailed trouble shooting. The combination of visual screens and automated scripts has greatly decreased the time required to identify and recover from problems associated with a large accelerator [3].

To help the operators develop their screens, an aggressive screen management system has been put into place [4]. Each operator can develop a prototype screen, but to get it approved for general use it must meet some basic criteria. Guidelines concerning color, size and naming conventions have been issued. In addition user feedback is very important; screens that are not being used are removed. While this may seem overly structured it does insure that the screens are consistent and that documentation is completed.

One of the largest and most complex systems to monitor is the RF system. It consists of 338 independently controlled superconducting accelerating cavities and 13 warm cavities for beam processing, chopping and separating. The expert screen for a superconducting cavity contains over 70 signals that are continuously monitored and is much too busy and complicated to use in day-to-day operations. One example of an operator developed screen is the RF status screen (figure 1) [3].

The screen in figure 1 displays the status of all the accelerating cavities (32) in an I/O controller. The square indicator for each cavity is a composite of nine signals that describe the condition of the cavity. The nine status signals are layered so that higher priority signals cover lower priority signals. The status screen also provides the operator with:

• The state of the high power amplifiers (HPA) that feed the eight cavities in each cryomodule.
• Buttons to access expert control screens for diagnosing problems in specific cavities.
• Buttons to access databases for documenting changes made to individual cavities.
• Reset buttons to clear common RF faults.

This screen is highly visible, and with a glance an operator can tell that something is wrong with the RF system. In most cases the operator will never need to go beyond this screen once the RF is operating.

If it is necessary to get to more detailed screens they are only a button away. The screens are treed in a fashion from lowest to highest in detail so you can continue to jump down to the screen with most functions and information. Controls screens for magnet, beam instrumentation and many other systems have been developed in a similar fashion.

At Jefferson Lab the brunt of the application programming fell upon the operational physicists [5]. They formed close working bonds with both the subsystem designers and the software programmers. In most cases applications were developed when it was necessary or would greatly enhance the commissioning process. Table 1 gives a detailed list of the programs developed and their functions.

Program	Function
Display Library	Assists in the building of display pages
Slow Feedback Locks	Keeps the beam at correct energy and orbit
On-Line Comments	Database established to assist in the RF maintenance
Autokrest	Automated program to phase cavities
Save/Restore Interface	Saves and stores operational set points
Linac Energy Management	Adjust cavities and quadrupoles to specific energy
Autosteer	Helps to steer beam through arcs and minimize orbit
On-Line Procedures	Operators can access procedures directly to X-terminals

Most of the high level applications were developed and prototype using the Tcl/Tk programming Toolbox [6,7]. The graphics user interface (GUI) is based on an interpretive shell called "wish", which is programmed in the Tcl language[7]. The Tcl language has a set of features geared toward event driven programming which is perfect for control systems where exceptions are more common than straight command flow. Tcl has worked well with the EPICS channel access routines. It can also be mated with computational software such as MatLab, numerical recipes, and optics codes such as DIMAD. We have extended the Tcl language by adding C compiled routines. By adding C routines the prototype programs have been made faster and more efficient and not reliant on the CPU thirsty Tcl language.

Slow Orbit and Energy Lock

The most important high level applications are those pertaining to energy and orbit lock. Energy and orbit lock are necessary since CEBAF is a mulitipass machine that has independent transport arcs for each pass. Therefore it is imperative that some form of feedback is utilized to insure the beam stays within the energy and orbit apertures. The ultimate requirement of the locks is to insure that the beam energy and energy spread are maintained, and to a lesser extent that no beam scraping occurs. The locks are also used for steering the beam to the proper orbit. This essentially makes the accelerator modular since a single lock sets the launch into each arc or pass.

The design approach has been one of eliminating the most pressing energy or orbit related drifts instead of meeting our stringent beam requirements right away. The locks that were first developed have been termed "slow locks" which eliminated most of the diurnal drifts and allowed the accelerator to be commissioned. The slow locks are used on a daily basis and without them the accelerator would only be able to function for short periods of time. Without the locks the accelerator would require expert attention from an operator many times an hour! By quickly developing the slow locks we were able to develop a more comprehensive fast lock system that has allowed CEBAF to meet our beam specifications without delays in initial operation.

The lock system uses beam position monitors (BPMs) to determine changes in energy and orbit [6]. Figure 2 shows the locations of the energy and orbit locks for both fast and slow locks. In the case of energy control a group of BPMs at high dispersion points in the arc look at the change in energy and then use the information to feed back on accelerating cavities in the linacs. The relative linac energy, DE/E, is controlled to better than using this method. Orbit lock uses several BPM and vertical and horizontal correctors to keep the beam centered in the accelerator. The bandwidth of the locks is approximately 0.2 Hz which is determined by the control system. Machine experts can select BPMs, corrector magnets, and RF cavities from the associated control panels and start the lock by toggling a button. Both energy and orbit lock programs make extensive use of a numerical matrix package. Matrices are sent to the package for singular value decomposition and the error vector is sent to the program every two seconds for back substitution.

As the machine was commissioned the locks have been upgraded to include more exception handling features for detecting beam loss and device failure. The locks made it possible to commission the accelerator by allowing operators to concentrate on new transport lines instead of continually retuning those already operational.

Fast Energy and Orbit Lock

To ultimately meet beam stability specifications for nuclear physics experiments, a fast feedback system has been developed [8]. Since the slow lock has a control bandwidth of only 0.2 Hz it cannot eliminate higher frequency noise signals. A fast feedback system is needed to correct for disturbances on beam position and energy at 60 and 180 Hz associated with AC line power harmonics. In addition noise components on beam energy have been observed at 4 and 10 Hz.

The fast feedback system has been based on the concepts of modern control theory using state space formalism [7]. Like the slow lock, BPMs in a high dispersion or orbit sensitive regions of the accelerator give a measurement of the beam's energy or betatron fluctuation. In the case of the energy lock the correction required is obtained by modulating the accelerating gradient in selected accelerating cavities upstream of the BPMs. The control input signal is calculated using the BPM measurements from the current sample and the state of the system from the previous sample. The controller design is based on a Linear Quadratic Gaussian controller/estimator and is optimized by using a system dynamics model, and process and noise statistics.

A prototype fast energy lock was installed in CEBAF's injector where energy variations are measured using five BPMs. To implement this fast energy lock two I/O controllers (IOCs) were remotely linked through Ethernet [8]. The BPM receivers are located in a CAMAC crate controlled by one of the IOCs. The correction signals are computed on this IOC, and sent via Ethernet to the other IOC which controls the RF. A CAMAC DAC card that resides close to RF control modules then sends out a 5 V signal to correct for the energy variations. The RF system has feedback inputs exclusively designed for this purpose. This prototype system was run at a 60 Hz rate.

Tests using the prototype system have shown that the system works very well. Figure 3 shows the effect of closing the loop on the energy lock in the injector [8]. An energy perturbation of 0.225 MeV was introduced upstream from the lock BPMs while the loop was open. The lock was then closed and the energy error was corrected. The system is very agile and can be operated using pulsed beam or with cw beam. In addition the system can also distinguish between betatron oscillations and energy variations. This prototype system has now been modified to perform measurement, computation of correction signal and actuation from a single IOC such that this system could be run at a 540 Hz rate.

Presently the fast lock system is being implemented as needed in the rest of the accelerator. The bandwidth of the full system will be scaled such that the 180 Hz noise component can be suppressed by 30 dB. The fast lock system will not make the slow locks obsolete. Rather they will complement each other such that the slow locks will take out the larger slow drifts, allowing the fast locks to have maximum headroom. Slow locks will also handle the detailed exception handling.

Linac Energy Management "LEM"

The ability to precisely set the energy of the two linacs is necessary to insure that the beam is within the energy aperture of all the transport arcs. A program affectionately known as LEM has been written to do this [6]. The program calculates the cavity gradients for each linac and then determines the set points for the quadrupoles in the linac to provide the desired FODO lattice. The program uses an optimized set of cavity gradients that limit the maximum cavity gradient due to: beam current, available RF power or cavity trip rates. During setup the program also compensates for non-crested and cavity calibration errors by adding a "fudge factor" as an input. The program is run whenever a cavity is removed or added during operations and for restoring the linacs after maintenance periods. Once the linac is LEM'd it is up to energy lock to maintain its energy.

A parasitic program that uses the LEM routine, "Kemcheck" calculates the headroom in each linac. It sums the requested cavity gradients and then compares them to the actual energy that is entering an arc. From this information an operator can not only determine headroom but also how well the cavities are phased. A large fudge factor and high gradient sum indicate poorly phased cavities.

Cavity Phasing

A novel use of the slow energy lock has been to phase (crest) all of the accelerating cavities in the two linacs. A program called Autokrest has been developed which uses the energy lock error signal to phase the cavities or crest the cavities for maximum acceleration [9]. The beam's energy spread is greatly dependent on the ability to crest the accelerating cavities [10]. The specification for slow phase drifts is 2.9^o rms for individual cavities. The energy locks do not differentiate between phase and amplitude drifts and do not take phase into account when compensating for energy loss. Therefore autokrest is needed to keep the cavities within phase specification to minimize the beam's energy spread. The program is also useful during accelerator startups when the cavities tend to be dephased due to maintenance activities. The program can be configured in a variety of ways to phase a whole linac, a cryomodule (8 cavities) or individual cavities. The program runs in the background transparent to beam operations. Operators have found that once the cavities have been phased it is only necessary to phase the cryomodules for periods exceeding 4 weeks.

The program works by shifting the phase of a cavity, cryomodule or linac by a prescribed amount and then observing the error signal of the slow energy lock BPMs. Typically a cavity's phase is shifted by , which corresponds to an energy shift of . From the energy shift it is quite easy to determine distance off phase crest, and the cavity is adjusted accordingly. The error in determining the crested phase set point is approximately 2^o. The crested phase resolution improves when the program is operated using groups of cavities like in a cryomodule.

Presently this program can be run continuously in the background even during experimental beam operations. The program is able to do this because energy lock is compensating for the phase shifts in the background. As long as the cavity is somewhat close to crest it will be adjusted. The program is not without its problems: other cavities can affect the measurement by perturbing the error signal while a cavity is being phased. Improvements to this are presently being implemented.

A good example of a subsystem application is the automated frequency tuning of the superconducting cavities. An operational problem for the CEBAF accelerator is the detuning of the cavities after a helium liquifier crash or after an extended maintenance period. The resonant frequency is very sensitive to pressure variations within the helium bath and changes of only a few mbar will detune the cavity. A program known as Autotune has been written that can tune a cavity as much as 5 kHz off from 1497 MHz [11]. The application lives above the RF cavity controllers in the cluster of HP9000-7xx workstations that control the RF. The program itself has three steps in tuning a cavity: a coarse tuning known as burst mode, a fine tuning known as sweep mode and finally a tracking mode that keeps the cavity tuned.

Burst mode is used to coarsely tune the cavity to within 160 Hz of resonance. It uses a pseudo-random phase modulated noise burst from the RF control module to measure the transfer function of the cavity. The modulated signal is such that its power spectrum is a positive constant for frequencies within 5 kHz and zero outside this range. By taking the Fourier transform of the cavity output, the resonance frequency can be determined. The bandwidth of the signal, 5 kHz, is limited by the RF cavity control module's microprocessor. For over 95% of the cavities this limited bandwidth has not been a problem. Cavities that are outside this range are initially tuned manually with a vector network analyzer.

Once the cavities are within 160 Hz they can be fined tuned with sweep mode. In this case the cavities are amplitude locked at a low gradient of 3 MV/m. A single sideband modulation is produced using the phase vector modulator. This modulation is then swept 200 Hz. A detuning angle offset is generated by comparing the forward power and the output of the cavity. The program uses smoothing techniques and curve fitting to extract parameters such as detuning angle offset and loaded Q and also to minimize the effect of microphonics. If it does not fit within expected limits the process is terminated and an alarm is generated. The control system uses the offset angle to normalize the RF controls to a resonance condition.

Finally a program called Autotrack is used to correct any differences from the normalized angle. The program automatically turns itself on if the detuning angles are greater than 10^o and turns off when the angle is within 3^o . This program also minimizes the over and undershoot of the stepper motor tuner due to the backlash and hysteresis of the gear train. Autotune has been succesfully working for almost two years with very few changes.

Any large process control whether, it is a petroleum refinery or a large particle accelerator, will depend on process and application specific programs. These programs are the real test of a control system. While all of the various subsystems - RF, beam instrumentation, magnets, etc.- have basically been tested for functionality and reliability, the control system is not tested until the commissioning and preliminary operations start. A control system must have the flexibility and speed to allow the multi-subsystem application programs to run. The high level programs at Jefferson Lab have been successful because the control system meets these basic criteria. EPICS is not without its detractors', it can be slow and new code upgrades have not always been bug free. This has been a handicap in some cases for the more complicated programs. The control system has been flexible enough so that solutions to these problems could be made with little or no impact on operations.

The success of CEBAF's control system can also be attributed to the working culture. It involves the close communication between the hardware and the software designers. Software engineers were brought in early on in the hardware development to assist in the design of prototype controls. In some cases they have developed a better feel for the functionality of the hardware than the designer! In a similar manner the accelerator physicists and operators have formed strong bonds with the control system experts to assist them in making the machine more efficient and easier to turn on. This is not to say there were no frustrations; there have been many, but they are easier to deal with in an atmosphere of pervasive mutual respect.

I am indebted to Dave Rauch and Sarah Spata, who assisted in the compilation of this paper.

[1] L. R. Dalesio, et al., "The Experimental Physics and Industrial Control System Architecture: Past, Present and Future", International Conference on Accelerator and Large Experimental Physics Control Systems, Oct. 1993.

[2] William A. Watson, "The CEBAF Control System", Proceedings of the 1995 Particle Accelerator Conference, May 1995.

[3] J. Karn, et al., "Operational Monitoring of the CEBAF RF System", Proceedings of the 1995 Particle Accelerator Conference, May 1995.

[4] T. Fox, private communication

[5] K. White and D. Kehne, "Operational Experience with CEBAF Control System", International Conference on Accelerator and Large Experimental Physics Control Systems, October 1995.

[6] J. van Zeijts, "Rapid Application Development Using Tcl/Tk Language", Proceedings of the 1995 Particle Accelerator Conference, May 1995.

[7] J. K. Ousterhout, "Tcl and the Tk Toolkit", Addison-Wesley Professional Computing Series (1996)

[8] M. P. Chowdhary, et al., " A Fast Feedback System For CEBAF", International Conference on Accelerator and Large Experimental Physics Control Systems, October 1995.

[9] M. G. Tiefenback and Kurt Brown, "Automated RF Cresting at CEBAF", CEBAF-TN-96-020

[10] G. Krafft et al., "Revised Energy Spread Estimate", CEBAF-TN-0050, May 1987.

[11] M. P. Chowdhary et al., "Automated Frequency Tuning of SRF Cavities at CEBAF", Proceedings of the 1995 Particle Accelerator Conference, May 1995.