generalTime

#####################################################################

Author: Babak Kalantari at PSI dot CH
Institution: Paul Scherrer Institute / SLS controls
Initial Version: 17.11.2006
Purpose: Description of usage and functionality of EventTime driver support for generalTime
Acknowledgment: The code was developed through interactive technical discussions with Timo Korhonen. Some parts of the code has been inspired by the earlier work of Jim Kowalkowsky.

#####################################################################

#########################################################################
###### modifictaion log #######
version 1.1
- initial version

version 1.2
- master PLL/validation relies only generalTime
- clock validation changed to aoRecord

The event time driver 'drvDevEventTime' has been developed to support hardware synchronized timestamps for generalTime driver. It provides tight clock synchronization among IOC's equipped with required hardware.
Great care has been taken into account to keep the code as OS-independent as possible, however, at present we can only gaurantee its proper operation under vxWorks. Where it was possible, we have used standard EPICS routines supported by libCom.

#########################################################################
##
### Organization of this document
##

• System requirements & configurations
• Relation to generalTime
• Initialization
• EPICS threads
• Hardware interface
• Clock monitoring & controls
• Data types

#########################################################################
##
### System requirements & configurations
##
- Epics 3.14.8 or higher.
- generalTime driver
- EventTime driver (drvDevEventTime.c)

For Event Master IOC:
- EVG
- EVR

For Event Slave IOC:
- EVR

There is no rule or limitation on the card number of EVG-EVR. The EventTime driver recognizes the first card (EVG/EVR) which is found, to be configured in the linked list of the EVG/EVR drivers as we have implemented this for EVG/EVR-100 series in the corresponding device drivers. This first card can have any card number for example 5.
To use the EventTime, the following configuration function should be called in the IOC startup script:

EvtTimeConfigure(
master, ## 0=slave, 1=master
sync_rate_sec, ## sync rate in seconds
clock_rate_hz, ## clock rate in Herz
master_port, ## master listens here (UDP port 18323)
slave_port, ## slave talks here (UDP port 18322)
time_out ## reply timout in ms
)

The event Master IOC is responsible for generation and distribution of the clock tick event (0x7c) and tick-counter-reset event (0x7d). The user has to make sure that the parameter "clock_rate_hz" is consistent with the physical tick frequncy which goes on the event stream. If the interface functions to control the tick rate (set/get_ClkFreq, see "Hardware Interface" section below) are present, the tick frequecny will be regulated to follow reference (global) time provided by generalTime otherwise tick rate stays constant and the EventTime still works but drifts from the global time during a long run (typically few weeks).

#########################################################################
##
### Relation to generalTime
##

The concept of generalTime introduced by Scheng Peng (SNS) provides a priority-based clock mechanism. To keep it simple, generalTime maintains a list of the time providers (clocks) each with a certain priority. Whenever asked for the current time the generalTime gets the time from the highest priority avaiable clock. Each time-provider has to register itself to generalTime at init using the registeration functions provided by generalTime driver. Each time provider is responsible for consistency of its own time and as soon as it finds out that it is not valid anymore should anounce itself as faulty. generalTime gets to know this by ckecking the return value of the call to time provider (epicsTimeERROR or notepicsTimeERROR). The EvenTime driver relation to generalTime also satisfies what was described. At its init routine EventTime driver checks all the requirements (see - initialization section below) and when everything turned out to be fine it registres itself with generalTime and declares itself as a time provider. There are some critical points where EventTime driver relies on generalTime:

1) when EventTime wants to setup its clock it has to set its time to an estimation of the current time (actual global time). It relies only on generalTime to provide a consitent time. At init EventTime asks generalTime by calling "epicsTimeGetCurrent()".

2) When the EventTime is invalid and the user (via aoRecord) tries to validate it, then a call to "epicsTimeGetCurrent()" gets the EPICS current time and the result is compared with the time read from a direct call to EventTime in order to judge if EventTime is ready to become valid.

#########################################################################
##
### Initialization
##

The initialization of EventTime is done via "EventTime_Init()" which is called at "initHookAtEnd" (timing hardware has to be initialized before EventTime).
The following sequence is done by the init routine:

1. checks if EventTime device private (pEventTimePvt) exists
2. EventTime is set to FALSE state (invalid)
3. checks if timing receiver HW (EVR) is installed
4. checks if the "get_ticks()" is installed (by the HW driver e.g. EVR driver)
5. checks if "ts_sync_signal()" is installed and uses it to get sync event number, if this function is not installed the "ER_EVENT_RESET_TICK" is used
6. checks if "force_sync()" is installed by HW driver and if not uses the "EVTforceSoftSync()" instead.
7. checks if there is a "direct_time()" provided
8. sets up the "event_table" buffer to maintain individual event timestamps
9. determines if it can have the role of Master/Slave according to the user request in configuration function and the info it has got in previous steps
10. EventTime registers its "EVTeventHandler()" to the receiver (EVR) driver
11. EventTime registers its "EVTerrorHandler()" to the receiver (EVR) driver
12. for the Master:
    - calls EVTstartStampServer()
    - calls EVTstartSyncServer()
    - calls EVTstartPLL() if "set_ClkFreq()" and "get_ClkFreq()" are present
13. for the Slave:
    - calls EVTsetClockFromMaster()
    - calls EVTstartSyncClient()
14. calls "EVTstartClkWatch()"
15. calls any HW specific initialization provided by HW driver (ts_drv_init)
16. registers itself to generalTime by a call to "generalTimeTpRegister()" with time provider priority set to 70

#########################################################################
##
### EPICS threads
##

• epicsThreadId EVTstartSyncServer();
  creates the server thread only on event Master IOC that broadcasts the sync time stamp every time a sync event (default 0x7d) is receive by the event system. The event time is not valid until the first sync event ocuuers. When the event time becomes invalid due to a reason the sync messages are not broadcast except if the detected fault is "EVT_ntpLock_lost". This is to inform slaves quickly to give up the master and invalidate their event time.
• epicsThreadId EVTstartStampServer();
  creates the Server thread only on event Master IOC that listens for time stamp and sync requests from slaves. If the event time is not valid the thread does not reply the request (to force slaves to give up the master).
• epicsThreadId EVTstartPLL();
  creates the server thread only on event Master IOC to keep event time in sync with a reference time provided by generalTime. It invalidates the event time when it cannot lock to refernce time and tells the slaves that it is faulty, but it tries endlessly to lock the refernce by restoring the initial frequency of the clock tick. When it succeeds to track reference time it validates the event time again. When EventTime is valid it gets the reference time by a calling generalTimeGetExceptPriority() which returns the best available time excluding a time provider (in this case EventTime).
• epicsThreadId EVTstartSyncClient();
  creates the client thread that listens for sync time stamp message (from Master) on a port and verifies with this, its sync time stamp. If it does not receive the sync message after three sync interval it invalidates the event time and sets the fault state to "EVT_sync_timeout". Furthermore if it receives a sync message where it says master has the state of "EVT_ntpLock_lost" then it also sets its fault state the same.
• epicsThreadId EVTstartClkWatch();
  creates a thread that periodically (2 seconds) checks if the current event time is in progress and if not, it invalidates the event time and sets the fault state to "EVT_tick_fail".

#########################################################################
##
### Hardware interface
##

Event time is only usable if adequate hardware (HW) is present in the system and the minimal requirements are supported and plugged (installed) in the following HW-interface structure. The event time driver checks this in its initialization routine.
/**
receiver
**/
An IOC which uses Event Time and is configured to be Slave, must set at least some members of the global variable 'TSrxHW' in dev/drv support of its timing hardware (e.g. event system) as following:

- card A non-negative integer corresponding to the reciver.
- have_rx A pointer to a function which recognizes the card (No specific functionality is required).
- get_ticks A pointer to a funtion which can give the actual value of the clock tick counter whenever is called.
- register_signal_handler A pointer to a registration function to provide event system indication to event time driver for event timestamp handling.
- register_error_handler A pointer to a registration function to provide event system indication to event time driver for error handling (e.g. borken link, lost heartbeat, etc.)
struct devTSrxHWStruct {
int numFuncs;
int card;
long (*have_rx)(int Card);
long (*get_ticks)(int Card, unsigned long *Ticks);
long (*register_signal_handler)(int Card, void(*func)());
long (*register_error_handler)(int Card, void(*func)());
long (*direct_time)();
long (*ts_drv_init)();
long (*ts_get_time)(struct timespec*);
long (*ts_user_get)(int event_number,struct timespec* sp);
long (*ts_sync_signal)(); /* gives sync event (default 0x7d) */
};
typedef struct devTSrxHWStruct devTSrxHW;
/**
transmitter
**/
An IOC which uses Event Time and is configured to be Master, in addition to set all the required members of TSrxHW (receiver), must also set the required members of global variable 'TStxHW' in dev/drv support of its timing hardware (e.g. event system) as following:

- card A non-negative integer corresponding to the transmitter (e.g. event generator).
- have_tx A pointer to a function which recognizes the card (No specific functionality is required).
- set_ClkFreq Pointers to functions which could set / get
- get_ClkFreq the freuqncy of the clock tick generator. This is used to provide the synchronization with external time refernce namely NTP server and compensatefor the drift rate. The clock tick generator can be a tick rate generator which is integrated in the event generator or a totally stand-alone device external to event generator (e.g. a pulse generator).

struct devTStxHWStruct {
int numFuncs;
int card;
long (*have_tx)(int Card);
long (*force_sync)(int Card);
long (*set_ClkFreq)(int Card, unsigned long freq);
long (*get_ClkFreq)(int Card, unsigned long *freq);
};
typedef struct devTStxHWStruct devTStxHW;

#########################################################################
##
### Clock monitoring & controls
##

- aoRecord support to validate the event time. When the EventTime has been invalidated for whatever reason (e.g. broken event stream), it should be validated after the problem has been solved. How this is handled is as the following: When the EventTime is invalid and the ao record is processed to validate it, record processing routine first asks generalTime (calling epicsTimeGetCurrent) for the current time; it immediately asks EventTime directly (calling EVTgetTimeStamp) too. Then it compares the difference of theses two with the VAL field and allows a validation of EventTime only when the differnce of the two time values is less than VAL seconds. Otherwise it rejects the validation operation and returns a WRITE alarm.
- mbbiRecord support to find out the reason of the event time invalidity (fauilor detection). The following are the possible states:
EVT_time_OK: event time is not faulty (it is valid)

EVT_stream_lost: occures when event system harware (receiver) detects that event stream is lost and sends an interrupt if enabled

EVT_sync_timeout: occures when the sysnchronization message broadcast from master does not arrive after 3*sync_rate_sec

EVT_ntpLock_lost: occures when Master is not able anymore to keep track of external time reference (e.g. NTP server)

EVT_tick_fail: occures when clock tick counter is not increasing

#########################################################################
##
### Data types
##


typedef enum { EVT_sync_master,
EVT_sync_slave,
EVT_direct_master,
EVT_direct_slave
} TStime_type;


typedef enum { EVT_time_OK,
EVT_stream_lost,
EVT_sync_timeout,
EVT_ntpLock_lost,
EVT_tick_fail} TSfault_type;

##############################################
#### fixes in iocClockRegister.c ####
##############################################
## located in
## "base-3.14.x/src/libCom/osi/os/vxWorks"
##
## The main point here is to allow generalTime to install the time
## providers by calling iocClockRegister() in drvGeneralTime.c
## Therefore the iocClockRegister() was modified as the following.
##
void iocClockRegister(pepicsTimeGetCurrent getCurrent,
pepicsTimeGetEvent getEvent)
{
if(!piocClockPvt) {
printf("iocClockRegister: iocClock NOT yet initialized\n");
iocClockInit();
}
piocClockPvt->getCurrent = getCurrent;
piocClockPvt->getEvent = getEvent;
if(piocClockPvt->getEvent==NULL)
printf("iocClockRegister: getEvent Not found! ...\n");
return;
}

##
## I added 'else' part to iocClockGetEvent() which is necessary
## when a record requests the event time upon processing; that
## means when the TSE field is set to a non-zero event number.
##
int iocClockGetEvent(epicsTimeStamp *pDest, int eventNumber)
{
if (eventNumber==epicsTimeEventCurrentTime) {
*pDest = piocClockPvt->clock;
return(0);
}
else {
epicsTimeGetEvent(pDest,eventNumber);
return(0);
}
return(epicsTimeERROR);
}
##############################################