FRIB SOLARIS Collaboration
The collaboration focuses on the development of the DAQ for the SOLARIS spectrometer.
Kinematic with DWBA Simulation
A web page is created for a simulation. Here
proposed schematics
The DAQ system will be contained within a private network provided by the Netgear nighthawk wifi router and a 16 ports 10Gb/s switch with a total data capacity of 320 Gb/s. The 16 ports switch is chosen for adding more digitizers in the future. The DAQ computer will be a rack server Dell R7525 with 2 CPU (total 32 core with 64 thread), 64 GB RAM, and a lot of storage. The primary mission of the DAQ computer is to control and readout the digitizer. It is also served as the database and achieves data server (also for background parallel data analysis). The DAQ will be protected by a UPS unit with 2700W and 8 outlets, sufficient to provide power for the DAQ computer, the Crate, the HV supply, and other devices. The DAQ computer (2U), the router (1U), the HV supply (8U), the Crate (8U), and UPS (2U) will be in one rack of 25U (StarTech.com 25U Open Frame Server Rack). Finally, there will be a Mac for remote control and analysis. I suggest a Mac studio (Apple M1 Max chip 10-core) and a 49-inch-wide screen monitor (Samsung 49-inch Odyssey G9).
Item | Config | size | Price |
---|---|---|---|
Rack server | Dell PowerEdge R755 | 2U | ~ $8000 |
AMD EPYC 3.0 GHz 16C/32T x 2 | |||
16 GB x 4 3200 GHz RAM | |||
12 (front) + 2 (rear) 3.5" HDD slots | |||
10Gb/s ethernet dual ports | |||
Mass storage | 16 TB HDD x 6 (Raid 6) = 64 TB + 32 TB fail-safe | ~ $2400 | |
HD tray | WORKDONE 12-Pack - 3.5" Hard Drive Caddy | ~ $200 | |
Temp storage | 8 TB SSD SATA | ~ $700 | |
19' rack | StarTech.com 25U Open Frame Server Rack | ~$300 | |
UPS | DELL EMC SmartUPS 3000 SMARTCONNECT 120V RM, 6 + 2 output | 2U | ~ $1900 |
Network | Netgear nighthawk wifi router + 16 x 10Gb/s, 320 Gb/s switch | 1U | ~$2400 |
Mac + monitor | Max studio (M1 10-core) + Samsung Odyssey G9 49-inch | ~$3000 |
Hardware
Item | Config | size |
---|---|---|
Rack server | Dell PowerEdge R7525 | 2U |
AMD EPYC 7302 3.0 GHz 16C/32T x 2 = 64 cores | ||
16 GB x 8 3200 GHz RAM = 128 GB | ||
12 (front) + 2 (rear) 3.5" HDD slots | ||
10Gb/s ethernet dual ports | ||
Mass storage | 2 X (16 TB HDD x 6 (Raid 10) = 48 TB + 48 TB fail-safe) | |
Temp storage | 8 TB SSD SATA | |
UPS | TRIPPLITE SU30000RTXLCD3U, 3000VA 2700W | 3U |
Network | 24-port PPoE 1Gb switch + 8-port 10Gb switch | |
Mac | Max studio (2023 version, M1 Ultra 20-core, 48-core GPU, 32-core Neural Engine, 64GB RAM, 4TB SSD) |
Rack Server
The rack server is a Dell PowerEdge R7525. It has total of 64 cores with 3.0 GHz.
storage
The OS disk is at one of the rear HD slot. It has 1.98 TB capacity and Ubuntu 22.04 was installed.
There is another slot at the rear. but it is using small from factor. We have a 8 TB SSD waiting.
12 x 16TB hard disks were installed in the front panel. They are divided into 2 groups, 6 for raw data storage, 6 for trace analysis. Both groups are RAID10 array. They have read/write speed of 326/394 MB/s. In theory, it can support 3 x 1 Gb/s in full speed.
OS, software, and services
An Ubuntu 22.04 is used for the OS.
Internal:SOLARIS Rack DAQ Setup
local network
The server has 2-port of 10Gb/s Ethernet. One is connected to a 8-port 10Gb/s switch for digitizers. another one is connect to the Mac studio.
Although each digitizer has 10Gb/s Ethernet port, the actual data rate would be far lesser. In present setting, 7 digitizers are connected to the 10Gb/s switch that the maximum data rate is 1.25GB/s.
There are 2 short PCI slots and 2 normal PCI slots for future extensions, such as 25Gb/s optical fiber ports, or additional 10Gb/s Ethernet ports.
FSU SOLARIS DAQ
The SOLARIS DAQ uses the 2nd generation CAEN digitizer VX2745. The communication library is totally different from the 1st generation digitizer, which makes the SOLARIS DAQ different from the FSU DAQ.
The FSU SOLARIS DAQ contains the following ingredients:
- Complete control of the CAEN 2nd digitizers VX2745 with PDD-PHA firmware.
- Scope for single channels, to display 2 analog traces and 4 digital traces.
- Data merging and sorting
- Connection to Database (optional)
- Connection to ELog (optional)
The DAQ program can be used as a general DAQ, not SOLARIS-oriented. However, it will come with
- SOLARIS-oriented control panel
- HV controller
- Target fan controller
And also (in a separate git repository)
Architecture
The main ingredient of the DAQ is the ClassDigitizer2Gen.h . It is a standalone C++ class for controlling and readout of a 2nd gen digitizer. It has methods specialized for DPP-PHA firmware. By using this class, a command-line DAQ can be developed. By having the digitizer controller separated from the GUI, it is easier to maintain.
The SOLARIS DAQ is using Qt6 for GUI development, after comparing CERN ROOT GUI, Gtk4, and IMGUI. IMGUI is fast and nice due to the GPU rendering, but it cannot be ssh and is limited by GPU. CERN ROOT GUI is actually using Qt4 with a ROOT modification. It provides a very good histogram, and scatter plots support, but it is not as rich as Qt6 for UI elements. Gtk4, the syntax is C-style, not C++, which is very alien to me. Although I use Qt6 for development, I did not use the Qt creator and the ui files for UI, it a simply code-based UI development.
The mainWindow.h is the, well, main window for the GUI.
The folder structure of the DAQ is shown in the figure on the right-handed side.
- the program's local folder -- this stores the programSettings.txt
- the raw data path -- this stores the raw data
- the analysis path -- this stores the expName.sh, which is updated by the DAQ for the runID, elogID, raw data path, and expName etc.
Analysis Path
The analysis folder should contain the following sub-folders
- working
- working/Settings
- working/Logs
Although the setting file will be saved with the raw data for each run, the working/Settings stores the custom settings files.
The Logs folder stores the Log msg. There are other logs msg from stdout, saved under a Logs folder inside the program's local folder.
Initial Settings of the DAQ after installation
Once the DAQ is installed and open, a main window is opened (see the left picture).
Two folders are required:
- Data Path and Root Data Path, these two paths can be the same.
- Analysis Path : this is the analysis code. The expName.sh will be stored inside. expName.sh saved the run number, expName, etc.
Before open digitizer, Click the Program Settings to define the paths, database IP and Name, and Elog IP. These settings will be saved in the Settings Save Path.
Data Reading and Structure
The new CAEN FeLib has two modes for data reading.
- Raw endpoint -- can get a chuck of data at 1 read-out. But the data needs to be decoded, and decoding requires time.
- DPP-PHA endpoint -- can only get 1 channel at a time, but the data is already decoded.
I tested the speed of the 2 methods. It turns out the DPP-PHA endpoint method is faster.
The syntax for getting these two endpoints are shown in the following.
Raw endpoint
uint64_t ep_handle;
CAEN_FELib_GetHandle(dev_handle, "/endpoint/raw", &ep_handle);
uint64_t ep_folder_handle;
CAEN_FELib_GetParentHandle(ep_handle, NULL, &ep_folder_handle);
CAEN_FELib_SetValue(ep_folder_handle, "/par/activeendpoint", "raw");
CAEN_FELib_SetReadDataFormat(ep_handle,
" [ \
{ \"name\": \"DATA\", \"type\": \"U8\", \"dim\": 1 }, \
{ \"name\": \"SIZE\", \"type\": \"SIZE_T\" }, \
{ \"name\": \"N_EVENTS\", \"type\": \"U32\" }, \
]"
);
uint8_t* data = new uint8_t[200000];
size_t size; /// number of byte of the data
uint32_t n_events;
CAEN_FELib_ReadData(ep_handle, 100, data, &size, &n_events );
DPP-PHA endpoint
uint64_t ep_handle;
ret = CAEN_FELib_GetHandle(dev_handle, "/endpoint/dpppha", &ep_handle);
//---------- configure endpoint
uint64_t ep_folder_handle;
ret = CAEN_FELib_GetParentHandle(ep_handle, NULL, &ep_folder_handle);
ret = CAEN_FELib_SetValue(ep_folder_handle, "/par/activeendpoint", "dpppha");
ret = CAEN_FELib_SetReadDataFormat(ep_handle,
" [ \
{ \"name\" : \"CHANNEL\", \"type\" : \"U8\" }, \
{ \"name\" : \"TIMESTAMP\", \"type\" : \"U64\" }, \
{ \"name\" : \"FINE_TIMESTAMP\", \"type\" : \"U16\" }, \
{ \"name\" : \"ENERGY\", \"type\" : \"U16\" }, \
{ \"name\" : \"ANALOG_PROBE_1\", \"type\" : \"I32\", \"dim\" : 1 }, \
{ \"name\" : \"ANALOG_PROBE_2\", \"type\" : \"I32\", \"dim\" : 1 }, \
{ \"name\" : \"DIGITAL_PROBE_1\", \"type\" : \"U8\", \"dim\" : 1 }, \
{ \"name\" : \"DIGITAL_PROBE_2\", \"type\" : \"U8\", \"dim\" : 1 }, \
{ \"name\" : \"DIGITAL_PROBE_3\", \"type\" : \"U8\", \"dim\" : 1 }, \
{ \"name\" : \"DIGITAL_PROBE_4\", \"type\" : \"U8\", \"dim\" : 1 }, \
{ \"name\" : \"ANALOG_PROBE_1_TYPE\", \"type\" : \"U8\" }, \
{ \"name\" : \"ANALOG_PROBE_2_TYPE\", \"type\" : \"U8\" }, \
{ \"name\" : \"DIGITAL_PROBE_1_TYPE\", \"type\" : \"U8\" }, \
{ \"name\" : \"DIGITAL_PROBE_2_TYPE\", \"type\" : \"U8\" }, \
{ \"name\" : \"DIGITAL_PROBE_3_TYPE\", \"type\" : \"U8\" }, \
{ \"name\" : \"DIGITAL_PROBE_4_TYPE\", \"type\" : \"U8\" }, \
{ \"name\" : \"WAVEFORM_SIZE\", \"type\" : \"SIZE_T\" }, \
{ \"name\" : \"FLAGS_LOW_PRIORITY\", \"type\" : \"U16\"}, \
{ \"name\" : \"FLAGS_HIGH_PRIORITY\", \"type\" : \"U16\" }, \
{ \"name\" : \"EVENT_SIZE\", \"type\" : \"SIZE_T\" } \
] \
"
);
uint8_t channel;
uint64_t timestamp;
uint16_t fine_timestamp;
uint16_t energy;
int32_t* analog_probes[2];
uint8_t* digital_probes[4];
analog_probes[0] = new int32_t[512];
analog_probes[1] = new int32_t[512];
digital_probes[0] = new uint8_t[512];
digital_probes[1] = new uint8_t[512];
digital_probes[2] = new uint8_t[512];
digital_probes[3] = new uint8_t[512];
uint8_t analog_probes_type[2];
uint8_t digital_probes_type[4];
size_t n_samples;
uint16_t flags_low_priority;
uint16_t flags_high_priority;
size_t event_size;
ret = CAEN_FELib_SendCommand(dev_handle, "/cmd/armacquisition");
ret = CAEN_FELib_SendCommand(dev_handle, "/cmd/swstartacquisition");
ret = CAEN_FELib_ReadData(ep_handle, 100,
&channel,
×tamp,
&fine_timestamp,
&energy,
analog_probes[0],
analog_probes[1],
digital_probes[0],
digital_probes[1],
digital_probes[2],
digital_probes[3],
&analog_probes_type[0],
&analog_probes_type[1],
&digital_probes_type[0],
&digital_probes_type[1],
&digital_probes_type[2],
&digital_probes_type[3],
&n_samples,
&flags_low_priority,
&flags_high_priority,
&event_size
);
Setting for Coincidence Trigger
This involve the following settings
setting | type | domain | example |
---|---|---|---|
ChannelsTriggerMask | byte | ch | ch-i is the i-bit or (1 << i), so, for ch-1 and ch-2, put 6. |
GlobalTriggerSource | string | dig | |
EventTriggerSource | string | dig | |
CoincidenceMask | ch | ||
AntiCoincidenceMask | ch | ||
example 1
Suppose there are 2 input signals to ch-1 and ch-24. And We want ch-24 to be recorded when ch-1 is triggered.
/ch/0..63/par/EventTriggerSource Ch64Trigger // the trigger is controlled by ChannlesTriggerMask /ch/0..63/par/WaveTriggerSource Ch64Trigger // for waveform recording /ch/1/par/ChannelsTriggerMask 0x2 // it is equal to ch-1 self-trigger. /ch/24/par/ChannelsTriggerMask 0x2 // ch-24 is triggered by ch-1
Notice that, ch-24 will be recorded no matter ch-24 has a signal or not.
example 2
Suppose there are 2 input signals to ch-1 and ch-23, And we want ch-23 to be recorded when ch-1 and ch-23 are both triggered.
/ch/1/par/EventTriggerSource ChSelfTrigger /ch/1/par/CoincidenceLengthT 40 /ch/23/par/EventTriggerSource ChSelfTrigger /ch/23/par/CoincidenceMask Ch64Trigger // the CoincidentMask is the ChannelsTriggerMask /ch/23/par/ChannelsTriggerMask 0x1
These will send the coincident window of 40 ns counted by ch-1. But I am not sure if is it before, after, or in the middle. ch-23 is a self-trigger, but it coincident with ch-1, so ch-23 only be recorded when the coincident is fulfilled.
Another setting can be
/ch/1/par/EventTriggerSource ChSelfTrigger /ch/23/par/EventTriggerSource ChSelfTrigger /ch/23/par/CoincidenceMask Ch64Trigger // the CoincidentMask is the ChannelsTriggerMask /ch/23/par/ChannelsTriggerMask 0x1 /ch/23/par/CoincidenceLengthT 40 // define the coincident window from ch-23
The 2nd example changes the CoincidentLength defined by ch-23.
(need to test and confirm)
example 3
Has 3 signals input at ch-1, ch-23, ch-45. We want ch-23 is recorded when ch-1 is trigger, but veto when ch-45 is trigger.
Signal of ch-23 only recorded when Trigger(ch-1) && !Trigger(ch-45)
Sync with multiple digitizers
Using the 4-pin clock/sync connector to connect the CLK-OUT (Master) to CLK-IN (Slave). Using Daisy chain for many slaves.
MASTER
CAEN_FELib_SetValue(dev_0_handle,'par/ClockSource’,'Internal'); CAEN_FELib_SetValue(dev_0_handle,'par/EnClockOutFP’,'True'); #enbale CLK-OUT
SLAVE
CAEN_FELib_SetValue(dev_1_handle,'par/ClockSource’,'FPClkIn'); CAEN_FELib_SetValue(dev_1_handle,'par/EnClockOutFP’,'True');
For the Start RUN signal distribution, there are different possibilities:
SW START / STOP
MASTER
CAEN_FELib_SetValue(dev_0_handle,'par/StartSource’,'SWcmd'); CAEN_FELib_SetValue(dev_0_handle,'par/SyncOutMode’,'Run');
SLAVE
CAEN_FELib_SetValue(dev_1_handle,'par/StartSource’,‘EncodedClkIn'); CAEN_FELib_SetValue(dev_1_handle,'par/SyncOutMode’,'SyncIn');
S-IN (external pulse edge) START / SW STOP
MASTER
CAEN_FELib_SetValue(dev_0_handle,'par/StartSource’,'SINedge'); CAEN_FELib_SetValue(dev_0_handle,'par/SyncOutMode’,'Run');
SLAVE
CAEN_FELib_SetValue(dev_1_handle,'par/StartSource’,‘EncodedClkIn'); CAEN_FELib_SetValue(dev_1_handle,'par/SyncOutMode’,'SyncIn');
S-IN (external pulse edge) START / STOP
MASTER
CAEN_FELib_SetValue(dev_0_handle,'par/StartSource’,'SINlevel'); CAEN_FELib_SetValue(dev_0_handle,'par/SyncOutMode’,'Run');
SLAVE
CAEN_FELib_SetValue(dev_1_handle,'par/StartSource’,‘EncodedClkIn'); CAEN_FELib_SetValue(dev_1_handle,'par/SyncOutMode’,'SyncIn');
FSU SOLARIS Analysis Package
Since the data format of the FSU SOLARIS DAQ is custom designed. An Event builder and the following analysis pipeline are also provided.
The code can be found at the FSU git repository : SOLARIS Analysis
The basic pipeline is following the [ https://github.com/calemhoffman/digios ANL HELIOS] code with much more improvements. ( it may no longer be true as the HELIOS code can/will be updated based on the SOLARIS code development)
Folder structure
Unlike the HELIOS code, where the daq and analysis are packed together, the daq and analysis are separated for SOLARIS. But the basic folder structure for the analysis are the same.
Analysis ├── SOLARIS.sh // It is recommended to source the SOLARIS.sh in the user .bashrc. so that the env variable SOLARISANADIR is there, and some alias like 2Working is there. ├── SetupNewExp // Switching or creating a new experiment. Its function is the same in the DAQ. ├── Cleopatra // Simulation package with the Ptolemy DWBA code. ├── armory // All the weapons are stored to analyze the data. The analysis code in this folder is not experiment specified. ├── data_raw // symbolic link to raw data ├── root_data // symbolic link to root_data └── working // All experimental specific files. This folder should contains all kind of configurations. ├── ChainMonitors.C // List of root files for data process ├── Mapping.h ├── Monitor.C/h ├── Settings // saved digitizers settings └── expName.sh // experimental name
Analysis Pipeline
The pipeline is shown in the left figure.
rawdata -- EventBuilder --> runXXX.root -- GeneralSort --> gen_runXXX.root -- Monitor.C --> histograms
Event Builder
The raw data is already time sorted and each digitizer has its own output files (split into 2 GB).
Using the SolReader Class, the raw data can be read block by block.
The output of the Event Builder is root file (name runXXX.root) that included a TTree and a TMacro for timestamp information.
The trace is stored as a fixed-length array of 2500.
GeneralSort
This applies the following to the runXXX.root
- Mapping.h
- Use TGraph to store the trace
- Support fitting of the trace
- Support multi-thread parallel processing.