FRIB SOLARIS Collaboration

The collaboration focuses on the development of the DAQ for the SOLARIS spectrometer.

Kinematic with DWBA Simulation

A web page is created for a simulation. Here

proposed schematics

The DAQ system will be contained within a private network provided by the Netgear nighthawk wifi router and a 16 ports 10Gb/s switch with a total data capacity of 320 Gb/s. The 16 ports switch is chosen for adding more digitizers in the future. The DAQ computer will be a rack server Dell R7525 with 2 CPU (total 32 core with 64 thread), 64 GB RAM, and a lot of storage. The primary mission of the DAQ computer is to control and readout the digitizer. It is also served as the database and achieves data server (also for background parallel data analysis). The DAQ will be protected by a UPS unit with 2700W and 8 outlets, sufficient to provide power for the DAQ computer, the Crate, the HV supply, and other devices. The DAQ computer (2U), the router (1U), the HV supply (8U), the Crate (8U), and UPS (2U) will be in one rack of 25U (StarTech.com 25U Open Frame Server Rack). Finally, there will be a Mac for remote control and analysis. I suggest a Mac studio (Apple M1 Max chip 10-core) and a 49-inch-wide screen monitor (Samsung 49-inch Odyssey G9).

Item	Config	size	Price
Rack server	Dell PowerEdge R755	2U	~ $8000
	AMD EPYC 3.0 GHz 16C/32T x 2
	16 GB x 4 3200 GHz RAM
	12 (front) + 2 (rear) 3.5" HDD slots
	10Gb/s ethernet dual ports
Mass storage	16 TB HDD x 6 (Raid 6) = 64 TB + 32 TB fail-safe		~ $2400
HD tray	WORKDONE 12-Pack - 3.5" Hard Drive Caddy		~ $200
Temp storage	8 TB SSD SATA		~ $700
19' rack	StarTech.com 25U Open Frame Server Rack		~$300
UPS	DELL EMC SmartUPS 3000 SMARTCONNECT 120V RM, 6 + 2 output	2U	~ $1900
Network	Netgear nighthawk wifi router + 16 x 10Gb/s, 320 Gb/s switch	1U	~$2400
Mac + monitor	Max studio (M1 10-core) + Samsung Odyssey G9 49-inch		~$3000

Hardware

Item	Config	size
Rack server	Dell PowerEdge R7525	2U
	AMD EPYC 7302 3.0 GHz 16C/32T x 2 = 64 cores
	16 GB x 8 3200 GHz RAM = 128 GB
	12 (front) + 2 (rear) 3.5" HDD slots
	10Gb/s ethernet dual ports
Mass storage	16 TB HDD x 6 (Raid 10) = 48 TB + 48 TB fail-safe
Temp storage	8 TB SSD SATA
UPS	TRIPPLITE SU30000RTXLCD3U, 3000VA 2700W	3U
Network	24-port PPoE 1Gb switch + 8-port 10Gb switch
Mac	Max studio (M1 Ultra 20-core, 48-core GPU, 32-core Neural Engine, 64GB RAM, 4TB SSD)

Rack Server

The rack server is a Dell PowerEdge R7525. It has total of 64 cores with 3.0 GHz.

storage

The OS disk is at one of the rear HD slot. It has 1.98 TB capacity and Ubuntu 22.04 was installed.

There is another slot at the rear. but it is using small from factor. We have a 8 TB SSD waiting.

12 x 16TB hard disks were installed in the front panel. They are divided into 2 groups, 6 for raw data storage, 6 for trace analysis. Both groups are RAID10 array. They have read/write speed of 326/394 MB/s. In theory, it can support 3 10Gb/s in full speed.

OS, softwares, and services

An Ubuntu 22.04 is used for the OS.

installed:

• Edge browser
• VS code

tobe install:

• CERN ROOT
• CAEN library
• grafana
• influxDB
• SOLARIS DAQ
• apache?

Internal:SOLARIS Rack DAQ Setup

local network

The server has 2-port of 10Gb/s Ethernet. One is connected to a 8-port 10Gb/s switch for digitizers. another one is connect to the Mac studio.

Although each digitizer has 10Gb/s Ethernet port, the actual data rate would be far lesser. In present setting, 7 digitizers are connected to the 10Gb/s switch that the maximum data rate is 1.25GB/s.

There are 2 short PCI slots and 2 normal PCI slots for future extensions, such as 25Gb/s optical fiber ports, or additional 10Gb/s Ethernet ports.

FSU SOLARIS DAQ

An Animation demonstrats the Settings Panel.

The SOLARIS DAQ uses the 2nd generation CAEN digitizer VX2745. The communication library is totally different from the 1st generation digitizer, which makes the SOLARIS DAQ different from the FSU DAQ.

The FSU SOLARIS DAQ contains the following ingredients:

• Complete control of the CAEN 2nd digitizers VX2745 with PDD-PHA firmware.
• Scope for single channels, to display 2 analog traces and 4 digital traces.
• Data merging and sorting
• Connection to Database (optional)
• Connection to ELog (optional)

The DAQ program can be used as a general DAQ, not SOLARIS-oriented. However, it will come with

• SOLARIS-oriented control panel
• HV controller
• Target fan controller

And also (in a separate git repository)

• Online analysis code

Architecture

Paths for SOLARIS DAQ.The database and Elog are optional.

The main ingredient of the DAQ is the ClassDigitizer2Gen.h . It is a standalone C++ class for controlling and readout of a 2nd gen digitizer. It has methods specialized for DPP-PHA firmware. By using this class, a command-line DAQ can be developed. By having the digitizer controller separated from the GUI, it is easier to maintain.

The SOLARIS DAQ is using Qt6 for GUI development, after comparing CERN ROOT GUI, Gtk4, and IMGUI. IMGUI is fast and nice due to the GPU rendering, but it cannot be ssh and is limited by GPU. CERN ROOT GUI is actually using Qt4 with a ROOT modification. It provides a very good histogram, and scatter plots support, but it is not as rich as Qt6 for UI elements. Gtk4, the syntax is C-style, not C++, which is very alien to me. Although I use Qt6 for development, I did not use the Qt creator and the ui files for UI, it a simply code-based UI development.

The mainWindow.h is the, well, main window for the GUI.

The folder structure of the DAQ is shown in the figure on the right-handed side.

• the program's local folder -- this stores the programSettings.txt
• the raw data path -- this stores the raw data
• the analysis path -- this stores the expName.sh, which is updated by the DAQ for the runID, elogID, raw data path, and expName etc.

Data Reading and Structure

The new CAEN FeLib has two modes for data reading.

1. Raw endpoint -- can get a chuck of data at 1 read-out. But the data needs to be decoded, and decoding requires time.
2. DPP-PHA endpoint -- can only get 1 channel at a time, but the data is already decoded.

I tested the speed of the 2 methods. It turns out the DPP-PHA method is faster.

The syntax for getting these two endpoints are shown in the following.

Raw endpoint

uint64_t ep_handle;
CAEN_FELib_GetHandle(dev_handle, "/endpoint/raw", &ep_handle);

uint64_t ep_folder_handle;
CAEN_FELib_GetParentHandle(ep_handle, NULL, &ep_folder_handle);
CAEN_FELib_SetValue(ep_folder_handle, "/par/activeendpoint", "raw");
 
CAEN_FELib_SetReadDataFormat(ep_handle, 
      " [ \
           { \"name\": \"DATA\",     \"type\": \"U8\", \"dim\": 1 }, \
           { \"name\": \"SIZE\",     \"type\": \"SIZE_T\" }, \
           { \"name\": \"N_EVENTS\", \"type\": \"U32\" }, \
        ]"
    );
 
uint8_t* data = new uint8_t[200000];
size_t  size;  /// number of byte of the data
uint32_t n_events;

CAEN_FELib_ReadData(ep_handle, 100, data, &size, &n_events );

DPP-PHA endpoint

uint64_t ep_handle;
ret = CAEN_FELib_GetHandle(dev_handle, "/endpoint/dpppha", &ep_handle);

//---------- configure endpoint
uint64_t ep_folder_handle;
ret = CAEN_FELib_GetParentHandle(ep_handle, NULL, &ep_folder_handle);
ret = CAEN_FELib_SetValue(ep_folder_handle, "/par/activeendpoint", "dpppha");
    
ret = CAEN_FELib_SetReadDataFormat(ep_handle,
      " [ \
          { \"name\" : \"CHANNEL\",              \"type\" : \"U8\"  }, \
          { \"name\" : \"TIMESTAMP\",            \"type\" : \"U64\" }, \
          { \"name\" : \"FINE_TIMESTAMP\",       \"type\" : \"U16\" }, \
          { \"name\" : \"ENERGY\",               \"type\" : \"U16\" }, \
          { \"name\" : \"ANALOG_PROBE_1\",       \"type\" : \"I32\", \"dim\" : 1 }, \
          { \"name\" : \"ANALOG_PROBE_2\",       \"type\" : \"I32\", \"dim\" : 1 }, \
          { \"name\" : \"DIGITAL_PROBE_1\",      \"type\" : \"U8\",  \"dim\" : 1 }, \
          { \"name\" : \"DIGITAL_PROBE_2\",      \"type\" : \"U8\",  \"dim\" : 1 }, \
          { \"name\" : \"DIGITAL_PROBE_3\",      \"type\" : \"U8\",  \"dim\" : 1 }, \
          { \"name\" : \"DIGITAL_PROBE_4\",      \"type\" : \"U8\",  \"dim\" : 1 }, \
          { \"name\" : \"ANALOG_PROBE_1_TYPE\",  \"type\" : \"U8\" }, \
          { \"name\" : \"ANALOG_PROBE_2_TYPE\",  \"type\" : \"U8\" }, \
          { \"name\" : \"DIGITAL_PROBE_1_TYPE\", \"type\" : \"U8\" }, \
          { \"name\" : \"DIGITAL_PROBE_2_TYPE\", \"type\" : \"U8\" }, \
          { \"name\" : \"DIGITAL_PROBE_3_TYPE\", \"type\" : \"U8\" }, \
          { \"name\" : \"DIGITAL_PROBE_4_TYPE\", \"type\" : \"U8\" }, \
          { \"name\" : \"WAVEFORM_SIZE\",        \"type\" : \"SIZE_T\" }, \
          { \"name\" : \"FLAGS_LOW_PRIORITY\",   \"type\" : \"U16\"}, \
          { \"name\" : \"FLAGS_HIGH_PRIORITY\",  \"type\" : \"U16\" }, \
          { \"name\" : \"EVENT_SIZE\",           \"type\" : \"SIZE_T\" } \
        ] \
      "
    );

uint8_t  channel;
uint64_t timestamp;
uint16_t fine_timestamp;
uint16_t energy;

int32_t*  analog_probes[2];
uint8_t*  digital_probes[4];

analog_probes[0] = new int32_t[512];
analog_probes[1] = new int32_t[512];

digital_probes[0] = new uint8_t[512];
digital_probes[1] = new uint8_t[512];
digital_probes[2] = new uint8_t[512];
digital_probes[3] = new uint8_t[512];

uint8_t  analog_probes_type[2];
uint8_t  digital_probes_type[4];

size_t   n_samples;
uint16_t flags_low_priority;
uint16_t flags_high_priority;
size_t   event_size;

ret = CAEN_FELib_SendCommand(dev_handle, "/cmd/armacquisition");
ret = CAEN_FELib_SendCommand(dev_handle, "/cmd/swstartacquisition");

ret = CAEN_FELib_ReadData(ep_handle, 100,
        &channel,
        &timestamp,
        &fine_timestamp,
        &energy,
        analog_probes[0],
        analog_probes[1],
        digital_probes[0],
        digital_probes[1],
        digital_probes[2],
        digital_probes[3],
        &analog_probes_type[0],
        &analog_probes_type[1],
        &digital_probes_type[0],
        &digital_probes_type[1],
        &digital_probes_type[2],
        &digital_probes_type[3],
        &n_samples,
        &flags_low_priority,
        &flags_high_priority,
        &event_size
      );