Data Server: Difference between revisions

From FSU Fox's Lab Wiki
Jump to navigation Jump to search
 
(12 intermediate revisions by the same user not shown)
Line 17: Line 17:
= Spec. =  
= Spec. =  
* Dell PowerEdge R540 Rack server
* Dell PowerEdge R540 Rack server
* CPU : Intel Xeon Gold, 36 cores
* CPU : Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz (18 core x 2 = 36 threads)
* RAM : 32 GB
* RAM : 4 X 32 GB (upgraded on May 20, 2024)
* Storage :  
* Storage :  
** 1.9TB SSD for OS and /home/ at the rear slot-1
** 1.9TB SSD for OS and /home/ at the rear slot-1
** 1TB for data (temporary) /mnt/data_1TB at the rear slot-2
** 8TB for data (temporary) /mnt/data_8TB at the rear slot-2
** 6x16TB for data /mnt/data0 at the front slot-1,2,3,4,5,6
** 6x16TB for data /mnt/data0 at the front slot-1,2,3,4,5,6
* OS : Ubuntu 20.04
* OS : Ubuntu 20.04
Line 66: Line 66:
|Memory Configuration Type || Performance Optimized || 370-AAIP
|Memory Configuration Type || Performance Optimized || 370-AAIP
|-
|-
|Memory DIMM Type and Speed || 3200MT/s RDIMMs || 370-AEVR
|Memory DIMM Type and Speed || 3200MT/s RDIMMs DDR4 || 370-AEVR
|-
|-
|Motherboard || PowerEdge R540 MLK Motherboard, V2 || 329-BFBK
|Motherboard || PowerEdge R540 MLK Motherboard, V2 || 329-BFBK
Line 113: Line 113:


{|class='wikitable'
{|class='wikitable'
! Software/Code !! Info.
! Software/Code !! Info. !! Run Command
|-
|-
| [https://root.cern/ CERN ROOT 6.26/02] || if you don't know, probably you are at the wrong place.
| [https://root.cern/ CERN ROOT 6.26/02] || if you don't know, probably you are at the wrong place. || root
|-
|-
| [https://sites.google.com/alumni.tsukuba.ac.jp/kshell-nuclear/ KSHELL] || Large-scale shell model calculation using multi-cores.
| [https://sites.google.com/alumni.tsukuba.ac.jp/kshell-nuclear/ KSHELL] || Large-scale shell model calculation using multi-cores. || kshell_ui.py
|-
|-
| PtolemyGUI || GUI for DWBA calculation using Ptolemy, it will create a file call DWBA.
| PtolemyGUI || GUI for DWBA calculation using Ptolemy, it will create a file call DWBA. || PtolemyGUI
|-
|-
| [https://jupyter.org/ Jupyter notebook] ||  
| [https://jupyter.org/ Jupyter notebook] ||  
Line 125: Line 125:
| [https://www.docker.com/ docker] ||
| [https://www.docker.com/ docker] ||
|-
|-
| [https://www.geany.org/ geany] || a nice editor, in additional to emacs, vim, and nano
| [https://www.geany.org/ geany] || a nice editor, in additional to emacs, vim, and nano || geany
|-  
|-  
| [https://geant4.web.cern.ch/support/user_documentation GEANT4 11.0.2] || installed path /usr/local, cmake with -DGEANT4_INSTALL_DATA=ON -DGEANT4_USE_OPENGL_XLL=ON
| [https://geant4.web.cern.ch/support/user_documentation GEANT4 11.0.2] || installed path /usr/local, cmake with -DGEANT4_INSTALL_DATA=ON -DGEANT4_USE_OPENGL_XLL=ON
Line 141: Line 141:
| [[gnuscope]] (via docker) || Only able to run at local, for user Tandem
| [[gnuscope]] (via docker) || Only able to run at local, for user Tandem
|-
|-
| Globus personal connect ||  
| Globus personal connect || || globusconnect
|}
|}


Line 151: Line 151:
| / || 1.9 TB || 533 MB/s || SSD || rear-1 || OS disk
| / || 1.9 TB || 533 MB/s || SSD || rear-1 || OS disk
|-
|-
| /mnt/data_1TB || 1 TB || 108 MB/s|| HDD || rear-2  || not recommended
| /mnt/data_8TB || 8 TB || - MB/s|| SSD || rear-2  || extra analysis storage
|-
|-
| /mnt/data0    || 80 TB || 340 MB/s || RAID-5 6x16TB HHD || front-1,2,3,4,5,6  || recommended
| /mnt/data0    || 80 TB || 340 MB/s || RAID-5 6x16TB HHD || front-1,2,3,4,5,6  || recommended
|-
| /mnt/data1    || 100 TB || 340 MB/s || RAID-5 6x20TB HHD || front-7,8,9,10,11,12  || installed on 2024-10-11
|}
|}


Line 216: Line 218:


</div></div>
</div></div>
= Globus Endpoint =
An endpoint is created for people to access the data.
UUID : 5ba06aee-87ff-11ef-8ebd-4177e580d0c6
Name : FSU Fox's Lab Data Server Ryan's End Point Shared
[the old end point with UUID : 99d0d880-2644-11ee-8193-195c41bc0be4 is deleted]


= Globus Personal Connect =
= Globus Personal Connect =
Line 245: Line 256:


Simply remove the folder '''~/.globusonline'''
Simply remove the folder '''~/.globusonline'''
==== To keep run in background ====
nohup /opt/globusconnectpersonal/globusconnect -start > output.log 2>&1 &


= Exp Folder format =
= Exp Folder format =
Line 383: Line 398:


for account and password, see [https://fsunuc.physics.fsu.edu/elog/dataServer/6 elog]
for account and password, see [https://fsunuc.physics.fsu.edu/elog/dataServer/6 elog]
= System monitoring =
I use Prometheus + node_exporter. Same in the [[Internal:Web_Server#Prometheus | FSUNUC ]] setup.


= Others =
= Others =

Latest revision as of 13:37, 11 October 2024

Notice Notice :   for requesting a user account, see Common Users Account
Photo of the rack of the data server. The data server is on the top. The achive data server is in the middle. The UPS is at the bottom.

The data server is built for storing all experimental raw data and for analysis. The primary data storage /mnt/data0 (72 TB, RAID 5 array from 6 16 TB disks), should be and only be for raw data storage. Frequent read-write data should be avoided at all costs because it will reduce the disk lifetime. Analysis source code and output should be stored in user's home directory. All necessary analysis codes/programs are provided and maintained by the system admin. Users can also request any programs to be installed or installed in their local.

Network IP and Alias

IP network alias Speed functions
128.186.111.42 pauli.physics.fsu.edu 1 Gb/s for user login
128.186.111.43 pauli-data.physics.fsu.edu 10 Gb/s for data transfer

Spec.

  • Dell PowerEdge R540 Rack server
  • CPU : Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz (18 core x 2 = 36 threads)
  • RAM : 4 X 32 GB (upgraded on May 20, 2024)
  • Storage :
    • 1.9TB SSD for OS and /home/ at the rear slot-1
    • 8TB for data (temporary) /mnt/data_8TB at the rear slot-2
    • 6x16TB for data /mnt/data0 at the front slot-1,2,3,4,5,6
  • OS : Ubuntu 20.04
Power of the DataServer.png

Detail Spec.

Here are the detail spec.
Items description Item number
Additional Network Cards On-Board LOM 542-BBBP
Additional Processor No Additional Processor 374-BBBX
Advanced System Configurations UEFI BIOS Boot Mode with GPT Partition 800-BBDM
Bezel No Bezel 350-BBBW
Dell EMC Luggage Tag 350-BBLI
Chassis Configuration 3.5"Chas up 12 HP HDs 2x3.5" Rear Drives,PERC/HBA11 321-BHDM
Dell Services: Hardware Support Basic Hardware Services: Business Hours (5X10) Next Business Day On Site Hardware Warranty Repair 3 Years 815-4327
Dell Hardware Limited Warranty Plus On-Site Service 815-4328
Deployment Services On-Site Installation Declined 900-9997
Embedded Systems Management iDrac9, Express 385-BBLC
Group Manager iDRAC Group Manager, Disabled 379-BCQY
Hard Drives 1TB 7.2K RPM SATA 6Gbps 512n 2.5in Hot-plug Hard Drive, 3.5in HYB CARR 400-ASHG
Hard Drives (PCIe SSD/Flex Bay) 1.92TB SSD SATA Read Intensive 6Gbps 512 2.5in Flex Bay AG Drive,3.5in HYB CARR, 1 DWPD 400-AXRY
Internal Optical Drive No Internal Optical Drive 429-AAIQ
Memory Capacity 32GB RDIMM, 3200MT/s, Dual Rank 16Gb BASE x8 370-AGDS
Memory Configuration Type Performance Optimized 370-AAIP
Memory DIMM Type and Speed 3200MT/s RDIMMs DDR4 370-AEVR
Motherboard PowerEdge R540 MLK Motherboard, V2 329-BFBK
Operating System No Operating System 619-ABVR
OS Media Kits No Media Required 421-5736
Password iDRAC,Factory Generated Password 379-BCSF
PCIe Riser 1xFH, 1 or 2CPU 330-BBHT
Power Cords Power Cord - C13, 3M, 125V, 15A (North America, Guam, North Marianas, Philippines, Samoa, Vietnam) 450-AALV
Power Management BIOS Settings Performance BIOS Settings 384-BBBL
Power Supply Single,Hot-plug Power Supply,495W 450-AGUG
Processor Intel Xeon Gold 5220 2.2G, 18C/36T, 10.4GT/s, 24.75M Cache, Turbo, HT (125W) DDR4-2666 338-BSDI
Processor Thermal Configuration 1 CPU Standard Thermal for 3.5" Chassis 322-BBBJ
Quick Sync 2 (At-the-box mgmt) Quick Sync 2 (At-the-box mgmt) 350-BBKV
Rack Rails ReadyRails Sliding Rails With Cable Management Arm 770-BBBR
RAID Configuration Unconfigured RAID 780-BCDS
RAID/Internal Storage Controllers PERC H750 Adapter, Low Profile 405-ABCC
Regulatory PowerEdge R540 CCC and BIS Marking, No CE Marking 389-DSXE
Shipping PowerEdge R540 Shipping 340-BSID
Shipping Information US Order 332-1286
Shipping Material PowerEdge R540 Shipping Material, V2 340-COSN
System Documentation No Systems Documentation, No OpenManage DVD Kit 631-AACK
Trusted Platform Module (TPM) Trusted Platform Module 2.0 461-AAEM

Software

Software/Code Info. Run Command
CERN ROOT 6.26/02 if you don't know, probably you are at the wrong place. root
KSHELL Large-scale shell model calculation using multi-cores. kshell_ui.py
PtolemyGUI GUI for DWBA calculation using Ptolemy, it will create a file call DWBA. PtolemyGUI
Jupyter notebook
docker
geany a nice editor, in additional to emacs, vim, and nano geany
GEANT4 11.0.2 installed path /usr/local, cmake with -DGEANT4_INSTALL_DATA=ON -DGEANT4_USE_OPENGL_XLL=ON
view3Dview for VMRL vituallization
python3-pip pip or pip3 are the command, avalible packages
/opt/Ptolemy/Cleopatra/Isotope.h a C++ Class (Isotope) for mass20.txt
(TBD) twoBodyReaction.h a C++ Class for A+a→b+B reaction kinematics.
(TBD) Radware
gnuscope (via docker) Only able to run at local, for user Tandem
Globus personal connect globusconnect

Storage and Disks

Disk Capacity Speed Type Slot Command
/ 1.9 TB 533 MB/s SSD rear-1 OS disk
/mnt/data_8TB 8 TB - MB/s SSD rear-2 extra analysis storage
/mnt/data0 80 TB 340 MB/s RAID-5 6x16TB HHD front-1,2,3,4,5,6 recommended
/mnt/data1 100 TB 340 MB/s RAID-5 6x20TB HHD front-7,8,9,10,11,12 installed on 2024-10-11

Common Users Account

tandem is the general user account.

Anyone wish to have a personal account can be contact Ryan mailto:rtang@fsu.edu, and provides following information

  • full name
  • username (the name for login)

[ admin only ] Create an user account

there is a bash script in /usr/bin/createNewUser.sh. This script will create an account and set disk-quota limit.

User Quota

The default quota for user is 50GB soft limit and 100GB hard limit on their home directory. The quota can be increase on request. User can also use the /mnt/data_1TB. The user quota for raw data saving at /mnt/data0 are unlimited.

  • soft limit: user still can use more than 50G in the next 7 days, after that, this 50G is the hard limit.
  • hard limit: user can never use more than 100G.

Setup

install the quota program
sudo apt install quota

edit the /etc/fstab

from this 

UUID=fc15d3a2-a49f-4974-8216-5949a740ef80 /   ext4    errors=remount-ro 0       1

to this: 

UUID=fc15d3a2-a49f-4974-8216-5949a740ef80 /   ext4    errors=remount-ro,usrquota 0       1

remount 

sudo mount -o remount /

check the new effect 

cat /proc/mounts | grep ' / '
/dev/sdc2 / ext4 rw,relatime,quota,usrquota</spam>,errors=remount-ro,stripe=64 0 0

enable the quota

sudo quotacheck -ugm /

This will create /aquota.user

check 

sudo quotaon -v /
/dev/sdc2 [/]: user quotas turned on

Set user quota

use the command 'setquota' to set user quota, in this example 50G is the soft limit, 100G is the hard limit. 

sudo setquota -u <User> 50G 100G 0 0 /

quota report 

sudo repquota -s /

Globus Endpoint

An endpoint is created for people to access the data. 

UUID : 5ba06aee-87ff-11ef-8ebd-4177e580d0c6
Name : FSU Fox's Lab Data Server Ryan's End Point Shared

[the old end point with UUID : 99d0d880-2644-11ee-8193-195c41bc0be4 is deleted]

Globus Personal Connect

Notice Notice :   Ryan already made an endpoint on the data server called FSU Fox's Lab Data Server Ryan's End Point

For more detail, please check https://docs.globus.org/how-to/globus-connect-personal-linux/

The program is stored in /opt/globusconnectpersonal/

to Run

~>/opt/globusconnectpersonal/globusconnect &

It will ask you to login, follow the steps and use your FSU credential.

enable access of data0

if you cannot access the data0, edit the file at you home ~/.globusonline/lta/config-path 

~/,0,1
/mnt/,1,1

The 1st line is the access of your home directory, not-share, allow read/write.

The 2nd line is the access of the /mnt/ directory, share, allow read/write.

To clear the setting / remove credential

Simply remove the folder ~/.globusonline

To keep run in background

nohup /opt/globusconnectpersonal/globusconnect -start > output.log 2>&1 &

Exp Folder format

Please use 

YYYY_MM_beam_reaction_PI_extra
  • YYYY : year in 4 digits
  • MM : month is 4 digits
  • beam : e.g. 12C
  • reaction : e.g. dp
  • PI : name of PI
  • extra : some extra comments, can be skipped.

For example, Eli performed 12C(d,n)reaction on April, 2022. 

2022_04_12C_dn_eli

Mount disk remotely

To mount the disk remotely, need the sshfs installed in the local machine. In Debian/Ubuntu 

~>sudo apt install sshfs

Create a mount directory in the local machine (say /mnt/dataRemote) 

~>sudo mkdir /mnt/dataRemote

Mounting 

~>sudo sshfs -o allow_other YYY@pauli-data.physics.fsu.edu:XXX  /mnt/dataRemote/

where YYY is the user account where XXX is the disk location in the data server, e.g. /mnt/data0

To mount on startup

edit /etc/fstab

add line 

ssh#YYY@pauli.physics.fsu.edu:XXX  /mnt/dataRemote/

Python3 packages

Here is the list of the python3 packages on May 6, 2022
  • apturl 0.5.2
  • bcrypt 3.1.7
  • blinker 1.4
  • Brlapi 0.7.0
  • certifi 2019.11.28
  • cffi 1.15.0
  • chardet 3.0.4
  • click 8.1.3
  • colorama 0.4.3
  • command-not-found 0.3
  • cryptography 36.0.2
  • cupshelpers 1.0
  • dbus-python 1.2.16
  • defer 1.0.6
  • distro 1.4.0
  • distro-info 0.23ubuntu1
  • duplicity 0.8.12.0
  • entrypoints 0.3
  • fasteners 0.14.1
  • future 0.18.2
  • globus-cli 3.5.0
  • globus-sdk 3.8.0
  • httplib2 0.14.0
  • idna 2.8
  • jmespath 0.10.0
  • keyring 18.0.1
  • language-selector 0.1
  • launchpadlib 1.10.13
  • lazr.restfulclient 0.14.2
  • lazr.uri 1.0.3
  • lockfile 0.12.2
  • louis 3.12.0
  • macaroonbakery 1.3.1
  • matplotlib 3.5.2
  • monotonic 1.5
  • netifaces 0.10.4
  • numpy 1.22.3
  • oauthlib 3.1.0
  • olefile 0.46
  • paramiko 2.6.0
  • pexpect 4.6.0
  • Pillow 7.0.0
  • pip 20.0.2
  • plotly 5.8.0
  • protobuf 3.6.1
  • pycairo 1.16.2
  • pycparser 2.21
  • pycups 1.9.73
  • PyGObject 3.36.0
  • PyJWT 2.3.0
  • pymacaroons 0.13.0
  • PyNaCl 1.3.0
  • pyRFC3339 1.1
  • PySimpleGUI 4.59.0
  • python-apt 2.0.0+ubuntu0.20.4.7
  • python-dateutil 2.7.3
  • python-debian 0.1.36ubuntu1
  • pytz 2019.3
  • pyxdg 0.26
  • PyYAML 5.3.1
  • reportlab 3.5.34
  • requests 2.22.0
  • requests-unixsocket 0.2.0
  • SecretStorage 2.3.1
  • setuptools 45.2.0
  • simplejson 3.16.0
  • six 1.14.0
  • ssh-import-id 5.10
  • systemd-python 234
  • typing-extensions 4.2.0
  • ubuntu-advantage-tools 27.7
  • ubuntu-drivers-common 0.0.0
  • ufw 0.36
  • unattended-upgrades 0.1
  • urllib3 1.25.8
  • usb-creator 0.3.7
  • wadllib 1.3.3
  • wheel 0.34.2
  • xkit 0.0.0

Uninterruptible Power Supply

Warning Warning :   shutdown protection not tested


A uninterruptible Power Supply (UPS) is used for power the data server, the Archive Data Server, and the Ethernet switches on the rack.

a program for monitoring the UPS is installed from https://www.cyberpowersystems.com/product/software/power-panel-business/powerpanel-business-4-8-1-linux/

The program provide a web interface. 

pauli.physics.fsu.edu:3052

for account and password, see elog

System monitoring

I use Prometheus + node_exporter. Same in the FSUNUC setup.

Others

Contact

Retrieved from "https://fsunuc.physics.fsu.edu/wiki/index.php?title=Data_Server&oldid=2472"