FRIB SOLARIS Collaboration

The collaboration focuses on the development of the DAQ for the SOLARIS spectrometer.

Kinematic with DWBA Simulation

A web page is created for a simulation. Here

proposed schematics

The DAQ system will be contained within a private network provided by the Netgear nighthawk wifi router and a 16 ports 10Gb/s switch with a total data capacity of 320 Gb/s. The 16 ports switch is chosen for adding more digitizers in the future. The DAQ computer will be a rack server Dell R7525 with 2 CPU (total 32 core with 64 thread), 64 GB RAM, and a lot of storage. The primary mission of the DAQ computer is to control and readout the digitizer. It is also served as the database and achieves data server (also for background parallel data analysis). The DAQ will be protected by a UPS unit with 2700W and 8 outlets, sufficient to provide power for the DAQ computer, the Crate, the HV supply, and other devices. The DAQ computer (2U), the router (1U), the HV supply (8U), the Crate (8U), and UPS (2U) will be in one rack of 25U (StarTech.com 25U Open Frame Server Rack). Finally, there will be a Mac for remote control and analysis. I suggest a Mac studio (Apple M1 Max chip 10-core) and a 49-inch-wide screen monitor (Samsung 49-inch Odyssey G9).

Item	Config	size	Price
Rack server	Dell PowerEdge R755	2U	~ $8000
	AMD EPYC 3.0 GHz 16C/32T x 2
	16 GB x 4 3200 GHz RAM
	12 (front) + 2 (rear) 3.5" HDD slots
	10Gb/s ethernet dual ports
Mass storage	16 TB HDD x 6 (Raid 6) = 64 TB + 32 TB fail-safe		~ $2400
HD tray	WORKDONE 12-Pack - 3.5" Hard Drive Caddy		~ $200
Temp storage	8 TB SSD SATA		~ $700
19' rack	StarTech.com 25U Open Frame Server Rack		~$300
UPS	DELL EMC SmartUPS 3000 SMARTCONNECT 120V RM, 6 + 2 output	2U	~ $1900
Network	Netgear nighthawk wifi router + 16 x 10Gb/s, 320 Gb/s switch	1U	~$2400
Mac + monitor	Max studio (M1 10-core) + Samsung Odyssey G9 49-inch		~$3000

Hardware

Item	Config	size
Rack server	Dell PowerEdge R7525	2U
	AMD EPYC 7302 3.0 GHz 16C/32T x 2 = 64 cores
	16 GB x 8 3200 GHz RAM = 128 GB
	12 (front) + 2 (rear) 3.5" HDD slots
	10Gb/s ethernet dual ports
Mass storage	2 X (16 TB HDD x 6 (Raid 10) = 48 TB + 48 TB fail-safe)
Temp storage	8 TB SSD SATA
UPS	TRIPPLITE SU30000RTXLCD3U, 3000VA 2700W	3U
Network	24-port PPoE 1Gb switch + 8-port 10Gb switch
Mac	Max studio (2023 version, M1 Ultra 20-core, 48-core GPU, 32-core Neural Engine, 64GB RAM, 4TB SSD)

Rack Server

The rack server is a Dell PowerEdge R7525. It has total of 64 cores with 3.0 GHz.

storage

The OS disk is at one of the rear HD slot. It has 1.98 TB capacity and Ubuntu 22.04 was installed.

There is another slot at the rear. but it is using small from factor. We have a 8 TB SSD waiting.

12 x 16TB hard disks were installed in the front panel. They are divided into 2 groups, 6 for raw data storage, 6 for trace analysis. Both groups are RAID10 array. They have read/write speed of 326/394 MB/s. In theory, it can support 3 x 1 Gb/s in full speed.

OS, software, and services

An Ubuntu 22.04 is used for the OS.

Internal:SOLARIS Rack DAQ Setup

local network

The server has 2-port of 10Gb/s Ethernet. One is connected to a 8-port 10Gb/s switch for digitizers. another one is connect to the Mac studio.

Although each digitizer has 10Gb/s Ethernet port, the actual data rate would be far lesser. In present setting, 7 digitizers are connected to the 10Gb/s switch that the maximum data rate is 1.25GB/s.

There are 2 short PCI slots and 2 normal PCI slots for future extensions, such as 25Gb/s optical fiber ports, or additional 10Gb/s Ethernet ports.

FSU SOLARIS DAQ

An Animation demonstrats the Settings Panel.

The SOLARIS DAQ uses the 2nd generation CAEN digitizer VX2745. The communication library is totally different from the 1st generation digitizer, which makes the SOLARIS DAQ different from the FSU DAQ.

The FSU SOLARIS DAQ contains the following ingredients:

• Complete control of the CAEN 2nd digitizers VX2745 with PDD-PHA firmware.
• Scope for single channels, to display 2 analog traces and 4 digital traces.
• Data merging and sorting
• Connection to Database (optional)
• Connection to ELog (optional)

The DAQ program can be used as a general DAQ, not SOLARIS-oriented. However, it will come with

A screenshot of the SOLARIS panel. This panel is customized by a Mapping file.

• SOLARIS-oriented control panel
• HV controller
• Target fan controller

And also (in a separate git repository)

• Online/offline analysis code

Architecture

Paths for SOLARIS DAQ.The database and Elog are optional.

The main ingredient of the DAQ is the ClassDigitizer2Gen.h . It is a standalone C++ class for controlling and readout of a 2nd gen digitizer. It has methods specialized for DPP-PHA firmware. By using this class, a command-line DAQ can be developed. By having the digitizer controller separated from the GUI, it is easier to maintain.

The SOLARIS DAQ is using Qt6 for GUI development, after comparing CERN ROOT GUI, Gtk4, and IMGUI. IMGUI is fast and nice due to the GPU rendering, but it cannot be ssh and is limited by GPU. CERN ROOT GUI is actually using Qt4 with a ROOT modification. It provides a very good histogram, and scatter plots support, but it is not as rich as Qt6 for UI elements. Gtk4, the syntax is C-style, not C++, which is very alien to me. Although I use Qt6 for development, I did not use the Qt creator and the ui files for UI, it a simply code-based UI development.

The mainWindow.h is the, well, main window for the GUI.

The folder structure of the DAQ is shown in the figure on the right-handed side.

• the program's local folder -- this stores the programSettings.txt
• the raw data path -- this stores the raw data
• the analysis path -- this stores the expName.sh, which is updated by the DAQ for the runID, elogID, raw data path, and expName etc.

Analysis Path

The analysis folder should contain the following sub-folders

• working
• working/Settings
• working/Logs

Although the setting file will be saved with the raw data for each run, the working/Settings stores the custom settings files.

The Logs folder stores the Log msg. There are other logs msg from stdout, saved under a Logs folder inside the program's local folder.

Initial Settings of the DAQ after installation

Main Window of the SOLARIS DAQ

Once the DAQ is installed and open, a main window is opened (see the left picture).

Two folders are required:

• Data Path and Root Data Path, these two paths can be the same.
• Analysis Path : this is the analysis code. The expName.sh will be stored inside. expName.sh saved the run number, expName, etc.

Before open digitizer, Click the Program Settings to define the paths, database IP and Name, and Elog IP. These settings will be saved in the Settings Save Path.

Data Reading and Structure

The new CAEN FeLib has two modes for data reading.

1. Raw endpoint -- can get a chuck of data at 1 read-out. But the data needs to be decoded, and decoding requires time.
2. DPP-PHA endpoint -- can only get 1 channel at a time, but the data is already decoded.

I tested the speed of the 2 methods. It turns out the DPP-PHA endpoint method is faster.

The syntax for getting these two endpoints are shown in the following.

Raw endpoint

uint64_t ep_handle;
CAEN_FELib_GetHandle(dev_handle, "/endpoint/raw", &ep_handle);

uint64_t ep_folder_handle;
CAEN_FELib_GetParentHandle(ep_handle, NULL, &ep_folder_handle);
CAEN_FELib_SetValue(ep_folder_handle, "/par/activeendpoint", "raw");
 
CAEN_FELib_SetReadDataFormat(ep_handle, 
      " [ \
           { \"name\": \"DATA\",     \"type\": \"U8\", \"dim\": 1 }, \
           { \"name\": \"SIZE\",     \"type\": \"SIZE_T\" }, \
           { \"name\": \"N_EVENTS\", \"type\": \"U32\" }, \
        ]"
    );
 
uint8_t* data = new uint8_t[200000];
size_t  size;  /// number of byte of the data
uint32_t n_events;

CAEN_FELib_ReadData(ep_handle, 100, data, &size, &n_events );

DPP-PHA endpoint

uint64_t ep_handle;
ret = CAEN_FELib_GetHandle(dev_handle, "/endpoint/dpppha", &ep_handle);

//---------- configure endpoint
uint64_t ep_folder_handle;
ret = CAEN_FELib_GetParentHandle(ep_handle, NULL, &ep_folder_handle);
ret = CAEN_FELib_SetValue(ep_folder_handle, "/par/activeendpoint", "dpppha");
    
ret = CAEN_FELib_SetReadDataFormat(ep_handle,
      " [ \
          { \"name\" : \"CHANNEL\",              \"type\" : \"U8\"  }, \
          { \"name\" : \"TIMESTAMP\",            \"type\" : \"U64\" }, \
          { \"name\" : \"FINE_TIMESTAMP\",       \"type\" : \"U16\" }, \
          { \"name\" : \"ENERGY\",               \"type\" : \"U16\" }, \
          { \"name\" : \"ANALOG_PROBE_1\",       \"type\" : \"I32\", \"dim\" : 1 }, \
          { \"name\" : \"ANALOG_PROBE_2\",       \"type\" : \"I32\", \"dim\" : 1 }, \
          { \"name\" : \"DIGITAL_PROBE_1\",      \"type\" : \"U8\",  \"dim\" : 1 }, \
          { \"name\" : \"DIGITAL_PROBE_2\",      \"type\" : \"U8\",  \"dim\" : 1 }, \
          { \"name\" : \"DIGITAL_PROBE_3\",      \"type\" : \"U8\",  \"dim\" : 1 }, \
          { \"name\" : \"DIGITAL_PROBE_4\",      \"type\" : \"U8\",  \"dim\" : 1 }, \
          { \"name\" : \"ANALOG_PROBE_1_TYPE\",  \"type\" : \"U8\" }, \
          { \"name\" : \"ANALOG_PROBE_2_TYPE\",  \"type\" : \"U8\" }, \
          { \"name\" : \"DIGITAL_PROBE_1_TYPE\", \"type\" : \"U8\" }, \
          { \"name\" : \"DIGITAL_PROBE_2_TYPE\", \"type\" : \"U8\" }, \
          { \"name\" : \"DIGITAL_PROBE_3_TYPE\", \"type\" : \"U8\" }, \
          { \"name\" : \"DIGITAL_PROBE_4_TYPE\", \"type\" : \"U8\" }, \
          { \"name\" : \"WAVEFORM_SIZE\",        \"type\" : \"SIZE_T\" }, \
          { \"name\" : \"FLAGS_LOW_PRIORITY\",   \"type\" : \"U16\"}, \
          { \"name\" : \"FLAGS_HIGH_PRIORITY\",  \"type\" : \"U16\" }, \
          { \"name\" : \"EVENT_SIZE\",           \"type\" : \"SIZE_T\" } \
        ] \
      "
    );

uint8_t  channel;
uint64_t timestamp;
uint16_t fine_timestamp;
uint16_t energy;

int32_t*  analog_probes[2];
uint8_t*  digital_probes[4];

analog_probes[0] = new int32_t[512];
analog_probes[1] = new int32_t[512];

digital_probes[0] = new uint8_t[512];
digital_probes[1] = new uint8_t[512];
digital_probes[2] = new uint8_t[512];
digital_probes[3] = new uint8_t[512];

uint8_t  analog_probes_type[2];
uint8_t  digital_probes_type[4];

size_t   n_samples;
uint16_t flags_low_priority;
uint16_t flags_high_priority;
size_t   event_size;

ret = CAEN_FELib_SendCommand(dev_handle, "/cmd/armacquisition");
ret = CAEN_FELib_SendCommand(dev_handle, "/cmd/swstartacquisition");

ret = CAEN_FELib_ReadData(ep_handle, 100,
        &channel,
        &timestamp,
        &fine_timestamp,
        &energy,
        analog_probes[0],
        analog_probes[1],
        digital_probes[0],
        digital_probes[1],
        digital_probes[2],
        digital_probes[3],
        &analog_probes_type[0],
        &analog_probes_type[1],
        &digital_probes_type[0],
        &digital_probes_type[1],
        &digital_probes_type[2],
        &digital_probes_type[3],
        &n_samples,
        &flags_low_priority,
        &flags_high_priority,
        &event_size
      );

Setting for Coincidence Trigger

This involves the following settings

setting	type	domain	example
EventTriggerSource	string	ch	define the trigger source for energy and time
WaveTriggerSource	string	ch	define the trigger source for wave form
ChannelsTriggerMask	bit mask	ch	ch-i is the i-bit or (1 << i), so, for ch-1 and ch-2, put 6.
GlobalTriggerSource	string	dig	This is a global trigger to trigger all channels
CoincidenceMask	string	ch	Set the coincident mask
AntiCoincidenceMask	string	ch	Set the veto
ConicidentLengthT	number	ch	the coincident length in ns

• Always record waveform samples and relative event parameters

Set ONLY the WaveTriggerSource

• Always record event parameters, and record the relative waveform samples only when there is space in the board memory buffer

Set BOTH WaveTriggerSource and EventTriggerSource

example 1 : Trigger by other channel

Suppose there are 2 input signals to ch-1 and ch-24. And We want ch-24 to be recorded when ch-1 is triggered.

/ch/0..63/par/EventTriggerSource   Ch64Trigger  // the trigger is controlled by ChannlesTriggerMask
/ch/0..63/par/WaveTriggerSource    Ch64Trigger  // for waveform recording
/ch/1/par/ChannelsTriggerMask      0x2          // it is equal to ch-1 self-trigger.
/ch/24/par/ChannelsTriggerMask     0x2          // ch-24 is triggered by ch-1

Notice that, ch-24 will be recorded no matter ch-24 has a signal or not.

example 2 : Trigger by coincident with other channel

Suppose there are 2 input signals to ch-1 and ch-23, And we want ch-23 to be recorded when ch-1 and ch-23 are both triggered.

/ch/1/par/EventTriggerSource     ChSelfTrigger 
/ch/23/par/EventTriggerSource    ChSelfTrigger 
/ch/23/par/CoincidenceMask       Ch64Trigger   // the CoincidentMask is the ChannelsTriggerMask
/ch/23/par/ChannelsTriggerMask   0x2
/ch/23/par/CoincidenceLengthT    40     // define the coincident window from ch-23

The 2nd example changes the CoincidentLength defined by ch-23.

The coincident window is defined as +/- the triggering time of ch-23. For example. if ch-23 is triggered at t = 100 ns. The coincident time window of 40 ns is starting from t = 60 ns to t = 140 ns. If ch-1 has a trigger within this window, ch-23 will be recorded.

I tested the coincident window by sending ch-1 with a 1 kHz pulse, and ch-23 with 50 kHz Poisson pulses.

example 3 : Anti coincident

Ch-18 is prohibited by the ch-0 trigger, but accept anything not coincident with ch-0.

/ch/0/par/EventTriggerSource     ChSelfTrigger 
/ch/18/par/EventTriggerSource    ChSelfTrigger 
/ch/18/par/CoincidenceMask       Disable   
/ch/18/par/AntiCoincidenceMask   Ch64Trigger 
/ch/18/par/ChannelsTriggerMask   0x1
/ch/18/par/CoincidenceLengthT    40

example 5 : Mutual coincident

Signals from ch-0 and ch-18 only be recorded when both channels are triggered.

We can use ITL to archive that.

ch-0/EventTriggerSource = Channel Self-Trigger
ch-0/Coincident Mask    = Channel 64 trigger
ch-0/Trigger Mask       = 0x40000000   # ch-30
ch-0/Coincident Length  = 104
ch-18/EventTriggerSource = Channel Self-Trigger
ch-18/Coincident Mask    = Channel 64 trigger
ch-18/Trigger Mask       = 0x1   # ch-1
ch-18/Coincident Length  = 104

I tested the setting, ch-0 and ch-18 have a coincident pulse with 10 Hz, pulse height of 23000 ch. And each of them has another uncorrelated 100 Hz pulse, pulse height of 7900 ch.

example 4 : Trigger other board

We have two boards this time, from board-0, ch-0, generate a TRG-OUT, the TRG-OUT is sent to TRG-IN in board-1, and triggers whatever channels.

in board-0, we can set the ch-0 to be part of the ITL-A. and TRG-OUT is ITL-A.
in board-1, set the coincident mask to be TRG-IN and the coincident length to be long enough.

Note that in the ITL-A Main Logic, if we pick AND, or Majority, the calculation is checked within one clock cycle, which is 62.5 MHz = 16 ns. If ch-0 and ch-18 are not both triggered within 16 ns, then no output of the ITL-A. That is why I picked the Main Logic to be OR.

Sync with multiple digitizers

Using the 4-pin clock/sync connector to connect the CLK-OUT (Master) to CLK-IN (Slave). Using Daisy chain for many slaves.

MASTER

CAEN_FELib_SetValue(dev_0_handle,'par/ClockSource’,'Internal');
CAEN_FELib_SetValue(dev_0_handle,'par/EnClockOutFP’,'True');    #enbale CLK-OUT

SLAVE

CAEN_FELib_SetValue(dev_1_handle,'par/ClockSource’,'FPClkIn');
CAEN_FELib_SetValue(dev_1_handle,'par/EnClockOutFP’,'True');

For the Start RUN signal distribution, there are different possibilities:

Method-1 : SW START / STOP

MASTER

CAEN_FELib_SetValue(dev_0_handle,'par/StartSource’,'SWcmd');
CAEN_FELib_SetValue(dev_0_handle,'par/SyncOutMode’,'Run');

SLAVE

CAEN_FELib_SetValue(dev_1_handle,'par/StartSource’,‘EncodedClkIn');
CAEN_FELib_SetValue(dev_1_handle,'par/SyncOutMode’,'SyncIn');

Method-2 : S-IN (external pulse edge) START / SW STOP

MASTER

CAEN_FELib_SetValue(dev_0_handle,'par/StartSource’,'SINedge');
CAEN_FELib_SetValue(dev_0_handle,'par/SyncOutMode’,'Run');

SLAVE

CAEN_FELib_SetValue(dev_1_handle,'par/StartSource’,‘EncodedClkIn');
CAEN_FELib_SetValue(dev_1_handle,'par/SyncOutMode’,'SyncIn');

Method-3 : S-IN (external pulse edge) START / STOP

MASTER

CAEN_FELib_SetValue(dev_0_handle,'par/StartSource’,'SINlevel');
CAEN_FELib_SetValue(dev_0_handle,'par/SyncOutMode’,'Run');

SLAVE

CAEN_FELib_SetValue(dev_1_handle,'par/StartSource’,‘EncodedClkIn');
CAEN_FELib_SetValue(dev_1_handle,'par/SyncOutMode’,'SyncIn');

FSU SOLARIS Analysis Package

Since the data format of the FSU SOLARIS DAQ is custom designed. An Event builder and the following analysis pipeline are also provided.

The code can be found at the FSU git repository : SOLARIS Analysis

The basic pipeline is following the [ https://github.com/calemhoffman/digios ANL HELIOS] code with much more improvements. ( it may no longer be true as the HELIOS code can/will be updated based on the SOLARIS code development)

Folder structure

Unlike the HELIOS code, where the daq and analysis are packed together, the daq and analysis are separated for SOLARIS. But the basic folder structure for the analysis are the same.

Analysis
├── SOLARIS.sh   // It is recommended to source the SOLARIS.sh in the user .bashrc. so that the env variable SOLARISANADIR is there, and some alias like 2Working is there. 
├── SetupNewExp  // Switching or creating a new experiment. Its function is the same in the DAQ. 
├── Cleopatra    // Simulation package with the Ptolemy DWBA code. 
├── armory       // All the weapons are stored to analyze the data. The analysis code in this folder is not experiment specified. 
├── data_raw     // symbolic link to raw data
├── root_data    // symbolic link to root_data
└── working      // All experimental specific files. This folder should contains all kind of configurations.
   ├── ChainMonitors.C     // List of root files for data process
   ├── Mapping.h           
   ├── Monitor.C/h       
   ├── Settings            // saved digitizers settings
   └── expName.sh          // experimental name

Analysis Pipeline

A flow diagram for the analysis pipeline

The pipeline is shown in the left figure.

rawdata -- EventBuilder --> runXXX.root -- GeneralSort --> gen_runXXX.root -- Monitor.C --> histograms

Event Builder

The raw data is already time sorted and each digitizer has its own output files (split into 2 GB).

Using the SolReader Class, the raw data can be read block by block.

The output of the Event Builder is root file (name runXXX.root) that included a TTree and a TMacro for timestamp information.

The trace is stored as a fixed-length array of 2500.

GeneralSort

This applies the following to the runXXX.root

1. Mapping.h
2. Use TGraph to store the trace
3. Support fitting of the trace
4. Support multi-thread parallel processing.