ZEUS Reconstruction Log: year 2000 (first processing)

 
Date Event
28-Jan-2000 Processing of year 2000 data starts, card file revision 3.39
Noscell00.list copied
Some jobs stop with "Candidate not in eselprobrank" in eem.fpp
31-Jan-2000 Update in eem.fpp cures "eselprobrank" problem.
  3-Feb-2000 Messages from MFUNPK appear: "FLTN/LFLTN mismatch"
Expert notified
New masks for failed FTD cells included. Card file revision 3.40, active from
run 35080.
  4-Feb-2000 MFUNPK problem has vanished, attributed to a "temporary acquisition 
problem" by expert.
  7-Feb-2000 Tobias fixes Lumi GAFs by remote diagnotics from Padova,  at least for runs 
35091 and 35099 
  8-Feb-2000 BPC South removed during shutdown. Next (DAQ) run 35178.
10-Feb-2000 Block of ZES variables (Dch_pt, Dch_eta, ...) found to be absent in new 
ZES ntuples. Experts are on CHEP in Padova.
11-Feb-2000 Processing halted.
13-Feb-2000 ZES ntuples bug fixed (zephyr/src/zes/zesbktpl.fpp) fixed.
Switched on generation of PMSTAT files.
14-Feb-2000 Processing resumes with run 35125. The output files are now called RDST2, 
MDST2, NTUP2, ... Runs before 35125 have to be reprocessed later.
15-Feb-2000 Runs 35105-35108 have bad CAL calibration. These runs are flagged as
"bad" for now, have to be reprocessed & flagged back later.
18-Feb-2000 "C5 vertex problem" observed. Explained by erroneous BPC timing information 
after removal of the south part on 8-Feb. 
23-Feb-2000 First indication of a "tape crisis". Even after intervention, only part of the tape
contingent asked for is supplied. This appears to be due to a 
tender for a big amount of tapes which was initiated in November but got stuck
in the administration. -> we receive only small amounts of tapes at a time. An
emergency order is on the way.
24-Feb-2000 New regular statistics process, recording daily event count taken & 
reconstructed.
28-Feb-2000 Updated ZEPHYR executable with "isles" fix. The number of permissible
isles has been increased to avoid job abends. Used from run 35342 onwards.
28-Feb-2000 Updated control cards, with masks for a complete layer in FTD2 & several more
cells in FTD1 & FTD2 (release 3.41). Used from run 35342 onwards.
 7-Mar-2000 Bprecon update (South part switched off) submitted to repository. Ulrich Fricke
finds out that Bprecon version used in 1999a.1 (=1998a.2) is not the latest 
one.
10-Mar-2000 End of tape crisis
Reprocessing of run 32701 stalled because of incomplete lumi GAF
LUMI327 updated.
14-Mar-2000 Histogram file reprocessed for run 35363.
16-Mar-2000 Histogram files reprocessed for runs 35387, 35396, 35420.
22-Mar-2000 New ZEPHYR control cards file with wider cuts for DST bit 11 (revision 3.42). 
Used from run 35552 onwards. 
27-Mar-2000 Runs 35575, 76, 80, 82(, 83), 84, 85 are lost on the logging machine. 
  2-Apr-2000 Oracle clients "oruninfo" and "rutil" fail on zenith platforms. The run database 
is not updated automatically anymore. Consultation with Oracle & network 
experts.
  5-Apr-2000 Harald Falkenberg finds & solves the problem on the database server. Pending 
updates go to run database.
17-Apr-2000 BPC timing correction removed in ccrecon. New executable linked with 
2000a release, used from run 35923 onwards.
Cell area in RTD (Layer 12,cells 100-107) masked in modified card input file
(revision 3.43), used from run 35923 onwards.
26-Apr-2000 Deletion of runs 35575, 76, 80, 82(, 83), 84, 85 from ep_runs requested.
27-Apr-2000 During reconstruction, O(2 events per run) are unreadable due to bad ZEBRA 
records eg in runs 35981-82. Possibly EVB problem. Informed U. Behrens.
28-Apr-2000 Increased CPU time (to ~8s) for jobs in run range 36000-36030. Both problems
vanish afterwards. According to Ulf, problem is related to bad CAL situation
around Good Friday, which led to huge number of cells & high deadtime.
 4-May-2000 Reconstruction interrupted because of power fail in computer center. Farm PCs 
had to be brought up manually. Still problems with AFS access on some machines 
afterwards.
14-Jun-2000 CTD reconstruction suffers from a modified gas mixture, which reduces the Lorentz
angle. In the track reconstruction, the Lorentz angle is hardcoded to 45 degrees.
Emergency GAFs are underway. Processing continues.
19-Jun-2000 Emergency GAFs are implemented.
19-Jun-2000 Power failure in computer center kills /recspool disk. Some monitor files, n-tuples
etc lost. Also zephyr_dbrec card files are lost.
20-Jun-2000 Spool disks are repaired.
21-Jun-2000 zephyr_dbrec card RCS file restored.
29-Jun-2000 Lumi GAF production was found to be stuck since 4 days. Data processing stalled.
This is during the collaboration week in Toronto.
  4-Jul-2000 Lumi GAF problem was traced to a hanging calilu process on the logger machine.
Production of Lumi GAFs takes ~1 day to catch up.
25-Jul-2000 H. Schnurbusch/I. Brock find RTD hole in layer 12 which should have been masked. 
in rev 3.43. It turns out that rev 3.43 of card file was incorrectly restored. File corrected 
and stored as revision 3.44.
Reconstruction with card file revision 3.44 starts with run 37197.
24-Aug-2000 Datataking completed, last run 37715.
  6-Sep-2000 CAL calibration for last runs complete.
10-Sep-2000 Processing of shifted vertex runs 37587-37639 completed.
11-Sep-2000 Last run of year 2000 processed.
15-Sep-2000 Reprocessing of early 2000 runs 35031-35121 finished, output RDST2/MDST2.
23-Sep-2000 Fix of DST bits in the run range 35179 - 35920 is starting. This is done on MDST
basis. First use of new reconstruction farm & RAID disks.