Archive Data Server: Difference between revisions

From FSU Fox's Lab Wiki
Jump to navigation Jump to search
No edit summary
 
(19 intermediate revisions by the same user not shown)
Line 1: Line 1:
The archive data server is build upon an relic Dell R710 rack server, which was bought around 2010 and haven't been using for long time. It was a format [[ANASEN]] DAQ computer. The server has 8 TB of storage.  
The archive data server is built upon a relic Dell R710 rack server, which was bought around 2010 and hasn't been used for long time. It was a former [[ANASEN]] DAQ computer. The server has 8 TB of storage.  


Users home directories and data and code of finished experiments will be stored in the server.  
User's home directories and data and code of finished experiments will be stored on the server.  


The directories are
The directories are
Line 14: Line 14:




= Solution on installing UEFI-boot OS =
= Solution on installing Modern OS =


This is a very old machines that boosting from BIOS that read only the 1st 512kB for the master boot record and the 2nd 512kB for the boottrap.img. It provides UEFI boot manual (so that Ubuntu 20 USB boot drive can run), but, most probably, the RAID controller dose not support the UEFI, so when boosting the RAID array from UEFI, it switches back to BIOS mode. That's why an installed Ubuntu 20.04 cannot boot and return the grub command line. In fact, in the grub command line, it only see 2 partitions, the (proc) and (hd0), which are both 512kB. This indicates that it is using BIOS mode to see the first 2 512kB for boot loader.  
Modern OS, like Ubuntu 20, use UEFI boot that is different from BIOS boot.


I first install Ubuntu 12.04. Able to boot, then update to Ubuntu 14.04. It works well. But after update to 16.04, boot error.
This is an old machine booting from BIOS that read only the 1st 512kB for the master boot record and the 2nd 512kB for the boottrap.img. It provides UEFI boot manual (so that Ubuntu 20 USB boot drive can run, with BIOD update, BIOS mode can also boot Ubuntu 20 USB), but, the RAID controller does not support the UEFI (in fact, it clearly shows that the Virtual Disk is supported by BIOS), so when boosting the RAID array from UEFI, it switches back to BIOS mode. That's why an installed Ubuntu 20.04 cannot boot and return the grub command line. In fact, in the grub command line, it only sees 2 partitions, the (proc) and (hd0), which are both 512kB. This indicates that it is using BIOS mode to see the first two 512kB for the boot loader.  


<s>In fact, we can install Lubuntu (or Ubuntu) 22.04 as legacy BIOS. We have to boot the startup drive in BIOS, then the Lubuntu will be installed as BIOS using master boot record. Ubuntu startup disk creator cannot do that. We use Rufus in Windows, setting the boot disk to be MBR (master boot record) and boot from BIOS.</s> tried, failed.
There is no way to bypass the RAID controller, as it is the only interface to connect hard disks to the motherboard.


The problem is the grub cannot read partition larger than 100 GB. In order to cope with that, we need to set the partition manually
The problem is the grub cannot read a partition larger than 100 GB. In order to cope with that, we need to set the partition manually
* 1 MB for BIOS_boot for master boot record, unformatted
* 1 MB for BIOS_boot (master boot record), unformatted
* 10 GB for /boot , et4
* 10 GB for /boot , ext4
* 8 GB swap
* 8 GB swap
* 500 MB for EFI System, FAT
* 500 MB for EFI System, FAT
* Rest /
* Rest /, ext4


Then Ubuntu 22.04 is able to installed and boot
Then Ubuntu 22.04 is able to installed and boot
=== an idea ===
#Install a PCIe card with NVMe M.2 SSD, and boot to the SSD?
#replace the CD-ROM with an SSD. The CD-ROM is using SATA cable to connect, power?


= Network =  
= Network =  
Line 44: Line 48:
RAM : 2 x 4 GB DDR3 RDIMM at 1067 MHz.
RAM : 2 x 4 GB DDR3 RDIMM at 1067 MHz.


RAID controller : PERC 6/i, PCIe 2.0 x 4, Data Transfer rate 3 Gb/s (SAS SFF-8484 32Pin), max disk size = 2 TB  
RAID controller : PERC 6/i, PCIe 1.0 x 8, Data Transfer rate 3 Gb/s (SAS SFF-8484 32Pin), max disk size = 2 TB  


storage: 4 x 2 TB RAID 5 = 6 TB
storage: 4 x 2 TB RAID 5 = 6 TB
Line 53: Line 57:


# a firmware upgrade to 4 TB (rumor? which firmware version at the present?)
# a firmware upgrade to 4 TB (rumor? which firmware version at the present?)
#: https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=f96nr&oscode=naa&productcode=poweredge-r710
# replace the PERC 6/i RAID controller with H700 (~ $65 from Amazon), which supports 6 TB. The cable also needs to change
#: see this post https://www.reddit.com/r/sysadmin/comments/2pjuy3/perc_6i_controller_with_6tb_drives/
Some people suggest that it is related to the sector size. The hard disk in the past used 512 bytes for a sector size and the partition table support 2^32 bit, so 2 TB is the max disk size. From the transition to 512 bytes to 4 KB sector size, there is a 512 emulation (512e). But it supports a 4 ~ 6 TB disk, not more.
using fdisk -l
~>sudo fdisk -l /dev/sda
Disk /dev/sda: 5.46 TiB, 5999532441600 bytes, 11717836800 sectors
Disk model: PERC 6/i
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


# replace the PERC 6/i RAID controller with H700, which supports 6 TB. Cable also need to change
indicate that it is using 512 bytes sector size (physical).
 
For the old 2 TB disk in [[Web Server|fsunuc]],
Disk /dev/sdb: 1.84 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: ST2000VN004-2E41
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


= Update BIOS =
= Update BIOS =
The origin BIOS version is 3.3.0. It does not allow boots from UEFI USB disk in BIOS mode. After upgrading to 6.6.0 it is OK.


#prepare an empty USB stick
#prepare an empty USB stick

Latest revision as of 16:24, 25 June 2022

The archive data server is built upon a relic Dell R710 rack server, which was bought around 2010 and hasn't been used for long time. It was a former ANASEN DAQ computer. The server has 8 TB of storage.

User's home directories and data and code of finished experiments will be stored on the server.

The directories are

directory purpose
/archiveData for archive data
/backupHome for backing up Home directory


Solution on installing Modern OS

Modern OS, like Ubuntu 20, use UEFI boot that is different from BIOS boot.

This is an old machine booting from BIOS that read only the 1st 512kB for the master boot record and the 2nd 512kB for the boottrap.img. It provides UEFI boot manual (so that Ubuntu 20 USB boot drive can run, with BIOD update, BIOS mode can also boot Ubuntu 20 USB), but, the RAID controller does not support the UEFI (in fact, it clearly shows that the Virtual Disk is supported by BIOS), so when boosting the RAID array from UEFI, it switches back to BIOS mode. That's why an installed Ubuntu 20.04 cannot boot and return the grub command line. In fact, in the grub command line, it only sees 2 partitions, the (proc) and (hd0), which are both 512kB. This indicates that it is using BIOS mode to see the first two 512kB for the boot loader.

There is no way to bypass the RAID controller, as it is the only interface to connect hard disks to the motherboard.

The problem is the grub cannot read a partition larger than 100 GB. In order to cope with that, we need to set the partition manually

  • 1 MB for BIOS_boot (master boot record), unformatted
  • 10 GB for /boot , ext4
  • 8 GB swap
  • 500 MB for EFI System, FAT
  • Rest /, ext4

Then Ubuntu 22.04 is able to installed and boot

an idea

  1. Install a PCIe card with NVMe M.2 SSD, and boot to the SSD?
  2. replace the CD-ROM with an SSD. The CD-ROM is using SATA cable to connect, power?

Network

IP : 128.186.111.52

Alias : newton.physics.fsu.edu

Spec.

Dell R710 rack server (11th gen, around 2010) The Hardware Manual is Here

CPU : Intel Xeon E5606 Quad-core @ 2.13 GHz x 2 = total 8 cores

RAM : 2 x 4 GB DDR3 RDIMM at 1067 MHz.

RAID controller : PERC 6/i, PCIe 1.0 x 8, Data Transfer rate 3 Gb/s (SAS SFF-8484 32Pin), max disk size = 2 TB

storage: 4 x 2 TB RAID 5 = 6 TB

OS : Ubuntu 22.04

solution to increase the supported disk size

  1. a firmware upgrade to 4 TB (rumor? which firmware version at the present?)
    https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=f96nr&oscode=naa&productcode=poweredge-r710
  2. replace the PERC 6/i RAID controller with H700 (~ $65 from Amazon), which supports 6 TB. The cable also needs to change
    see this post https://www.reddit.com/r/sysadmin/comments/2pjuy3/perc_6i_controller_with_6tb_drives/

Some people suggest that it is related to the sector size. The hard disk in the past used 512 bytes for a sector size and the partition table support 2^32 bit, so 2 TB is the max disk size. From the transition to 512 bytes to 4 KB sector size, there is a 512 emulation (512e). But it supports a 4 ~ 6 TB disk, not more.

using fdisk -l 

~>sudo fdisk -l /dev/sda
Disk /dev/sda: 5.46 TiB, 5999532441600 bytes, 11717836800 sectors
Disk model: PERC 6/i
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

indicate that it is using 512 bytes sector size (physical).

For the old 2 TB disk in fsunuc, 

Disk /dev/sdb: 1.84 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: ST2000VN004-2E41
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Update BIOS

The origin BIOS version is 3.3.0. It does not allow boots from UEFI USB disk in BIOS mode. After upgrading to 6.6.0 it is OK.

  1. prepare an empty USB stick
  2. have a windows PC
  3. download the RUFUS
  4. download the latest BIOS 6.6.0 https://dl.dell.com/FOLDER05012843M/1/R710-060600C.exe
  5. use the window PC, run RUFUS
    1. select FreeDOS
    2. Check the 2nd, 3rd and 4th check boxes
    3. make the bootable disk
  6. in Dell R710, F11 to boot manual (if UEFI, switch to BIOS)
  7. Select "Hard Drive C:"
    1. Select USB
  8. then it will go to DOS
  9. run the R710-060600C.exe
  10. follow the instruction
Retrieved from "https://fsunuc.physics.fsu.edu/wiki/index.php?title=Archive_Data_Server&oldid=1117"