A couple of preservation workflows (such as full system preservation through imaging) or processing in digital forensics depend on reliable hardware-software stacks for identity system disk migrations. As especially the x86 platform is moving forward very fast, the hardware and software changes rapidly. Even if the standard suggests compatibility, there are a number of pitfalls which might prevent authentic copies of the original components to an image file.
Despite the formerly wide spread use of (parallel) port IDE disks in x86 computers, there are a couple of compatibility issues with these devices. These issues might prevent the full system preservation (as e.g. described in Full Disk Preservation of a MS-SQL database) to fail partially or completely. While e.g. the 40pin the physical interface did not change, the logical layer did. In the early days limitations of the system BIOS prevented older machines to properly "see" larger disks. For full system preservation the situation is usually reverted: The IDE port implementation might be significantly newer than the disk to be dumped.
Experimental Setup
Parallel port IDE was gradually phased out and replaced by SATA in the mid 2000. Thus, this kind of connector is not found in new x86 machines, which illustrates just another problem of hardware obsolescence. Older machines were found that still had this kind of connection. In the experiments several different IDE implementations (out of the stock of hardware available) were used:
- Intel 865 chipset with a BIOS from 2005 (compact PC with Pentium processor)
- Intel 875 chipset with a BIOS from 2004 (device not available any more because of died hardware)
- Two port onboard controller (Silicon Image parallel port adaptor, same mainboard with Intel 875)
- NVidia nForce chipset of 2005 (AMD CPU system)
- Lindy cable multi (physical) IDE port to USB 2.0 adaptor
- Davicontrol PCI dual-port IDE adaptor (external PCI card, Silicon Image PCI0680 Ultra ATA 133/166)
The identical IDE disks primariy considered where 240 MByte Quantum disks of an experiment to preserve a digital research environment featuring a DB2 database as a core component. The disks, taken from five client machines (the server was equipped with a SCSI disk which did not produce any hassles and identity of the imaging was easily proven), where numbered 1 … 5. Other disks tested where 13, 30, 40 GByte disks dating from around the last turn of the century. For the system imaging procedure a fairly recent 3.2 Linux kernel (Ubuntu 12.04.2 LTS) was used.
Fortunately we got a set of identical (regarding production date and size) IDE disks to test. The partition layout was roughly the same with some variance. The installed operating system was OS/2 version 2.1 on OS/2 HPFS, a filesystem which is directly mountable in Linux.
First Step – Disk Recognition
A preliminary requirement to disk reading is the proper recognition of the device with the controller. This got checked with the various Gigabyte disks without any problems, but produced partly "interesting" results for the 20 year old disks:
- The USB adaptor "saw" the disk, but was unable to produce a proper capacity reading (guessed on 2 TByte)
- Newer IDE adaptors, like e.g. the onboard controller did not recognize the disk at all
- Several disks were recognized by the Intel 865, 875 controllers, but two failed (#2, #4) in some earlier experiment
- The failed disk 4 was properly recognized on the nForce and Davicontrol ports
Nevertheless, the old disks were not properly recognized in every boot-up cycle. They needed a certain time to spin up to answer the BIOS/operating system requests properly. Sometimes they "hung", which is indicated by a permanent lighted on-disk activity LED. To see, if the disks were recognized by the operating system (Linux), the kernel messages give the information on which disks are visible to the system. Later on, run with administrator privileges, the fdisk command
should give a proper listing of the contained partitions. The recognition was somewhat tricky:
- Disk 2 spins up but does not get detected on any
- Disk 3 is only detected on the Davicontrol
- Most of the disks were not detected with every bootup. Pausing the machine, rebooting usually helps to get it finally done.
- If a diskis not properly recognized during bootup, then the
- unloading and loading of the IDE controller kernel module triggers the recognition. This usually helps.
- The detection rate on the Davicontrol adaptor was different in different machines. The BIOS and succession of devices in the bus (different order of initialization) seems to influence the process (significantly)
Second Step – Disk Reading
The tool of choice to produce identical copies of block devices in Linux/Unix systems is dd. In standard configuration it reads the block device 512 Byte-wise and writes this to a file (if asked to). After proper recognition the disk was present through the high-level device, e.g. /dev/sda and through the device for each partition e.g. /dev/sda1…5 (numbering depending on the partitions detected).
Every disk aside disk 3 was read from beginning to end with dd if=/dev/sda of=image-file. This procedure copies every thing including the master boot record as well as the partition table. dd finished the process without any errors in every run on every machine. The machine log did not show any errors either. Thus, it was concluded, that the process ran flawlessly. Unfortunately, the simple partition check on the image file fdisk -l image-file did not necessary produce the proper partition listing. This was assumed to be a deficiency of the tool. After trying to boot the resulted system image in emulator or virtual machine, severe mounting problems with the contained filesystems occured (especially with the system partition). Investigating into that revealed that different dd runs (with different blocksize settings tested) produced different image files (using the diff utility and md5sum). Unfortunately no errors were reported from the system at all. Fortunately, by chance the fdisk run on the original medium produced different listings, including incomplete or corrupted partition tables, looking like:
Disk /dev/sda: 245 MB, 245426688 bytes 13 heads, 51 sectors/track, 723 cylinders, total 479349 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sda1 204867 476033 135583+ 5 Extended /dev/sda2 * 51 204866 102408 7 HPFS/NTFS/exFAT /dev/sdb3 * 476034 478685 1326 a OS/2 Bootmanager /dev/sdb5 ? 4143585081 3991997998 2071690107 f6 Unknown Partition table entries are not in disk order omitting empty partition (6)
The original reading should have produced:
Disk /dev/sda: 245 MB, 245426688 bytes 13 heads, 51 sectors/track, 723 cylinders, total 479349 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sda1 204867 476033 135583+ 5 Extended /dev/sda2 * 51 204866 102408 7 HPFS/NTFS/exFAT /dev/sdb3 * 476034 478685 1326 a OS/2 Bootmanager /dev/sdb5 * 204918 426308 110695+ 7 HPFS/NTFS/exFAT /dev/sdb6 * 426360 476033 24837 7 HPFS/NTFS/exFAT Partition table entries are not in disk order
Other examples of exactly the same disk showed different patterns for the /dev/sdb5 entry line.
This can be cross-checked with hexdump. Different readings produce different results already in the first couple of Kilobytes. Unfortunately the failures are not easy to be distinguished from proper ones. A proper partition table listing from the image file does not necessarily ensure a complete success. Without the proper tools and insight into the reading results a proper proof is difficult to produce as any non-failed dd reading looks correct. The experiences from numerous other tests never hinted on problems to be expected. The dumped images behaved as expected (booted in the emulator without any block device specific errors). As many of the test used non-intrusive dumping, it at least ensured that hardware specific errors could be ruled out (because the machine worked properly with the original operating system). The only source of errors could be the dumping mini Linux in these cases.
The only disk with real reading errors (because of hardware defects, not just producing wrong output) was disk 3:
[ 457.107249] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 457.107275] ata3.00: failed command: READ SECTOR(S)
[ 457.107293] ata3.00: cmd 20/00:03:2d:08:00/00:00:00:00:00/a3 tag 0 pio 1536 in
[ 457.107295] res 51/40:02:2d:08:00/00:00:00:00:00/a3 Emask 0x9 (media error)
[ 457.107317] ata3.00: status: { DRDY ERR }
[ 457.107328] ata3.00: error: { UNC }
[ 457.152340] ata3.00: configured for PIO2
[ 457.152362] sd 2:0:0:0: [sda] Unhandled sense code
[ 457.152366] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 457.152372] sd 2:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[ 457.152378] Descriptor sense data with sense descriptors (in hex):
[ 457.152382] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[ 457.152396] 00 00 15 7d
[ 457.152402] sd 2:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
[ 457.152410] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 15 7d 00 00 03 00
[ 457.152424] end_request: I/O error, dev sda, sector 5501
[ 457.152445] Buffer I/O error on device sda, logical block 5501
[ 457.152460] Buffer I/O error on device sda, logical block 5502
[ 457.152472] Buffer I/O error on device sda, logical block 5503
The error listing clearly states problem reading sector 5501+. Such error messages usually coincide with getting an error with dd or problems reading the filesystem in a partition.
To rule out a faulty version of the used Linux kernel, the experiments were repeated with similar results on a three year older kernel with similar results. To check for implementation flaws in the IDE driver even older versions were booted, but then the hardware did not get fully recognized and a disk dump was impossible.
Image Verification
After getting repeatedly different results from reading the disks with the i865 chip set, other devices were looked at for dumping. The AMD/nForce system from about the same era as the i865 system behaved pretty much the same but was able to read disk 4 which was un-accessible on the i865 (reason not totally clear: We may not have tried hard enough to register the disk to the machine by restarting, delaying bootup or reloading the required kernel modules). After a couple of runs, fdisk produced different results (which indicated an early error at the very beginning of the disk) for the same disk, same for the dd runs. The produced images were different again from the images produced on the i865 system. The Davicontrol adaptor was a recent addition to the hardware pool and was used next. Up to now it has never failed to produce a proper partition table reading of different disks. The test was repeated regularly. The produced images were exactly the same in different runs, tested with diff and md5sum. The partition listing of the resulted image files looked as expected (identical to the reading from the original source). The same was true for mounting the partitions (none of the previous filesystem errors encountered from the other images). Thus, it should be pretty obvious that the setup is producing the expected results and has none of the hardware issues of the setups used before. Nevertheless, there might be still hidden errors around.
Conclusions
dd does exactly the job it is expected to, imaging block devices block wise to image files. If the input is corrupted for some reason (but not reported as an error detected by the operating system), dd (or similarly dd_rescue) has no means to detect such an issue. The verification process of the system imaging is to be done thoroughly using different tools and methods to ensure correctness. Rerunning, and comparing the results is one of the available options. The encountered errors highlight once again the importance of a hardware archive offering various options to try on the device to be imaged. Adaptors and system buses get out of use and result in device obsolescence which might prevent the proper preservation of a system. Non-intrusive disk dumping, which requires a suitable dumping OS paired with proper hardware available, at least circumvents the problems detected in this case study.
