GIS3 CPU trouble
An e-mail from Dr. Maxima and the GIS team on Apr 20, 1994
From firstname.lastname@example.org Wed Apr 20 06:02:18 1994 Received: by legacy.gsfc.nasa.gov (5.65/DEC-Ultrix/4.3) id AA22983; Wed, 20 Apr 1994 06:09:42 -0400 Date: Wed, 20 Apr 94 19:05:34 +0900 From: email@example.com (Kazuo Makishima) Message-Id: <9404201005.AA25436@miranda.phys.s.u-tokyo.ac.jp> To: firstname.lastname@example.org Status: RO Dear Asca Colleagues: ************************************************************************ * We regret that we must inform you a trouble in one of the two GIS * * CPU memories, of which you may have already heard from Nagase-san * * via e-mail. The trouble caused the pulse hieght data of GIS-S3 to be * * edited somewhat in a wrong shape. This lasted from February 10 till * * April 8, for a length of two months. Below we describe the nature of * * the trouble, its solution, its cause, its impact upon the GO data * * acquired in this period, and proposed action for prevention of * * similar problems in future. * ************************************************************************ 1. The Trouble On April 9, we were noticed by the KSC duty scientists that the routine software for monitoring the GIS gain fails to fit the GIS-S3 isotope line. A quick investigation at U. Tokyo revealed that the S3 pulse height spectrum is in a wrong shape: normally the spectrum is output in 1024 (10 bits) channels covering up to 12 keV, but the spectrum obatined exhibited events only every 8 channels. Apparently the lower three bits of the pulse height information was fixed at  (or 5 in decimal) for all the S3 events. A backword search indicated that the problem persisted for a considerable length of time, without being noticed by anybody. We now understand that the trouble happened on February 10, between 22:05 and 22:35 UT, after passing through the SAA. The target being observed was AWM7. The reason why the problem was not noticed for two months is twofold. One is that the event rate is rather low, so that the spectrum obtained in the KSC quick look usually contains too few counts to reveal the problem. The other is of courese that the trouble happened in the GO phase, when the data could not be accessed by the hardware team. 2. Solution Such a trouble can be caused either by the hardware failure of the ADC (analog-to-digital converter) or related circuit elements in GIS-E, or by the software error. To discriminate between these two possibilities, on April 8 (first pass occured at JST 21:59) we performed GIS memory check (MEMCHK), and telemetered the contents of the GIS memory (two 32kB RAMs) down to the ground. Note that GIS MEMCHK can be done by a single discrete command, and takes 32 seconds. We found one errornous word in the CPU3 memory. The word was immediately corrected from KSC by a block command (called RAM-patch action), and the S3 pulse height spectrum returned normal. We admit that another problem happened during these operations, namely the rise time (RT) spectrum of S3 became strange. However this is nothing to do with the hard-wired processing, and was solved until the 5th contact (in early morning of April 9). For simplicity we will skip on this subject today. 3. Cuause of the Trouble On analyzing the memory-check results, we found that the faulty 8bit word was in the program area of the CPU3 memory. It exactly corresponds to the address pointer, pointing to the address where lower 5 bits of the hard-wire processed 12-bit PH information and lower 2 bits of the S3 event timing is stored. (Note that the GIS uses 12-bit ADCs, but we always discard lower 2 bits to get 10-bit PH information.) As this word was destroyed, the CPU was reading in some irrelevant information from some wrong (but harmless in view of the CPU operation) address. This completely explaines what had happened for the two months. The GIS memory has 4-bit error correction code (Hamming code), and 1-bit errors must be corrected by the CPU. However if 2-bit error occurs in a single word, the CPU will no longer operate normally. Actually the faulty word discovered in the memory check was [C230] in hex, while it should normally be [C201]. Therefore this is a 3-bit error in a word. This is very likely to be caused initially by a 2-bit error, i.e. [C201]-->[C231]. Then after an automatic Hamming-code correction, [C231] is modified into [C230]. In summary, a 2-bit error happened in a word in the CPU3 memory on February 10, probably within the SAA. This error did not cause the CPU hang-up, but instead made the CPU3 to read wrong information as to lower 5 (effectively lower 3) bits of the S3 PH information, as well as lower 2 bits of the S3 timing information. 4. Impact upon the GO Data The impact of the trouble on the GO data acquired between February 10 and April 9 is twofold. First, the lower 3 bits of the PH output information for all the S3 events are wrong. However if we bin up the 1024-ch (10 bit) of PH into 128-ch (7 bit), the resulting spectrum is completely normal. We therefore believe that the S3 data can still be utilized, especially for faint sources or sources without significant spectral feature. Another impact is on the timing information. The GIS events can be tagged with up to 10 bits of timing information in non-standard operation, and the lower 2 bits are lost due to the trouble. This do not affect any time information in normal-mode operation, or high-time-resolution data with timing bits equal to or less than 8 bits. As far as we are aware, there are a few observations during the period conducted with 10-bit time information. It has yet to be examined wheter the loss of lower 2 bits is fatal or not to these observations. 5. Future Prevention of Similar Troubles It seems that 2-bit error occurs more frequently than is inferred from the ocurrence rate of the 1-bit errors, although we do not yet know whether this is specific to GIS or not. Since the 2-bit error occurred on the logically adjacent bits, we suspect that it is not due to a chance coincidence of two 1-bit errors, but rather due to a single impact by an energetic particle. For the prevention of similar problems in future, we will caryy out the following two actions. Firstly, we have asked the operation taem to conduct the GIS memory check typically once every day. If errors are found, the KSC duty scientists and operators will immediately perform the RAM-patch action to correct the errors. We believe that this has already started at KSC. The other is to conduct the quick-look at University of Tokyo, typically once or twice a week, to see any strange behavior in the data. This means that a very small fraction of the GO data will be stored at Univ. Tokyo, but off course, we will not do any scientific research. Finally we, the GIS team, aplogize to all the colleagues for this problem and any inconvenience that might be caused by it. We would appreciate your thoughtful understanding. Regards, April 20 K. Makishima and M. Tashiro : University of Tokyo (email@example.com, firstname.lastname@example.org) T. Ohahsi : Tokyo Metroploitan University (email@example.com) M. Ishida : ISAS (firstname.lastname@example.org) and the GIS team
If you have any questions concerning ASCA, visit our Feedback form.