Pixie16 digitizer: Difference between revisions
| Line 334: | Line 334: | ||
once downloaded the program. Unless the PlxSDK and PixieSDK are in difference location, using make should able to compile the program. | once downloaded the program. Unless the PlxSDK and PixieSDK are in difference location, using make should able to compile the program. | ||
There is only 1 configuration file <b>Pixie16.conf</b> | There is only 1 configuration file <b>[[#Pixie16.conf]]</b> | ||
To run, | To run, | ||
Revision as of 17:46, 25 April 2022
Introduction
In order to control the Pixie16 digitizer, the DAQ computer needs to recognize the digitizer via a PCIe card( for example NI PCI-8366).
The digitizer crate (XIA PXI CompactPCI) has 14-slot. The 1st slot must be the communication card (for example, NI PXI-8366). The remaining 13 slots can be used for digitizer and other modules.
The PCIe card and PXI Board requires no driver. The DAQ computer will treat it as a bridge.
In the computer (LINUX), using the command lspci to show the PCI connection
>lspci
as an example, part of the output:
04:00.0 PCI bridge: Intel Corporation 41210 [Lanai] Serial to Parallel PCI Bridge (A-Segment Bridge) (rev 09) 04:00.2 PCI bridge: Intel Corporation 41210 [Lanai] Serial to Parallel PCI Bridge (B-Segment Bridge) (rev 09) 05:0f.0 Unassigned class [ff00]: National Instruments PXI-8368 06:05.0 PCI bridge: Texas Instruments PCI2050 PCI-to-PCI Bridge (rev 02) 06:0f.0 Bridge: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge (rev 0b)
In the above output, the NI PXI-8368 board on the crate is detected. Also the PCI9054 communication chip is detected.
The PCI9054 (or PLX9054) chip is located at the corner of the Pixie16 digitizer.
The PCI9054 chip requires a driver. The driver is provided by Broadcom, and the package is called PlxSDK.
After the driver for the PCI9054 chip is done, the DAQ PC can talk to the digitizer. Next, the PixieSDK has to be installed for controlling the digitizer. The PixieSDK provides methods to control the digitizer.
Versions of Pixie16 Digitizer
In FSU, we have 100MHz (sampling rate) models and 250 MHz models.
500 MHz models are also used in Clarion2.
Each model has 16 channels. Using connector ?? </red>
Data Structure
The recorded data is stored in the extern FIFO (first-in first-out) memory or ExtFIFO on the digitizer. A block of Data is formed by word, each word is 32 bit long. A minimum data block contains 4 words. When extra information is switched on, they will be stored afterward. The total length of a data block is called event length.
The data structure shown below and taken from the XIA Pixie-16 User Manual version 3.06, page 72.
The trace (if enabled) is stored after the header.
Incomplete data block when retrieving
It is possible that an incomplete data block may be retrieved at the beginning or at the end of ExtFIFO.
If the data is saved continuously, that will be not a problem.
Broadcom PlxSDK
The chip was a product of PLX Technology, acquired by Broadcom Inc in 2014. The chipset driver is now called the Broadcom PCI/PCIe Software Development Kits [1]. The package provides complete documentation and driver. The last release is on 2020.
Because of that, it only support Linux kernel around that time. I tested on
| OS | Status |
| Debian 10 | OK |
| Debian 11 | Error |
| Ubuntu 20 | Error |
installation
After downloaded the package, unzip it, and we have
├── Documentation │ ├── PLX API DLL with Visual Basic.htm │ ├── PLX_LegacyAPI.pdf │ ├── PlxRdkReferenceGuide.htm │ ├── PLX_SDK_General_FAQ.pdf │ ├── PLX_SDK_Linux_Release_Notes.htm │ ├── PLX_SDK_Release_Notes.htm │ └── PlxSdkUserManual.pdf └── PlxSdk.tar
The tarball contains the driver. I extract the tarball into /usr/opt/PlxSdk. To setup the package (make) and also make the driver:
~>cd /usr/opt/PlxSdk/ PlxSdk>export PLX_SDK_DIR=$(pwd) PlxSdk>sudo make # This shold make everything, including things in Samples PlxSdk>cd Driver PlxSdk/Driver>export PLX_CHIP=9054 PlxSdk/Driver>sudo ./builddriver 9054 PlxSdk>cd ../Bin PlxSdk/Bin>sudo ./Plx_load 9054 Install: Plx9054 Load module......... Ok (/usr/opt/PlxSdk/Driver/Source.Plx9000/Output) Verify load......... Ok Get major number.... Ok (MajorID = 243) Create node path.... Ok (/dev/plx) Create nodes........ Ok (/dev/plx/Plx9054)
In the last step, it loads the 9054 driver to the Linux Kernel. Fro more detail, see | elog
Load 9054 driver on start up
To make the driver get loaded on start up, create a file at /etc/systemd/system/, say broadcom.service
[Unit] Description=Broadcom PCI/PCIe 9054 Driver After=network.target [Service] Type=oneshot Environment=PLX_SDK_DIR=/usr/opt/PlxSdk/ ExecStart=/bin/bash /usr/opt/PlxSdk/Bin/Plx_load 9054 ExecStop=/bin/bash /usr/opt/PlxSdk/Bin/Plx_unload 9054 RemainAfterExit=yes [Install] WantedBy=multi-user.target
and then
sudo systemctl daemon-reload sudo systemctl enable broadcom.service
Pixie SDK
PixieSDK provides all methods to control the digitizer, from parameters setting, writing/reading control register, starting and stopping the DAQ, to retrieving recorded data from its Extern FIFO (first-in-first-out) memory.
PixieSDK 3.2
The SDK can be downloaded from GitHub in here.
The methods are listed in this web page.
PixieSDK 3.3
PixieSDK 3.3 is similar to 3.2.
Legacy PixieSDK
DAQ Programs
NSCL DAQ (DDAS)
Pixie16 digitizer is supported by NSCL DAQ via the DDAS (digital data acquistion system) package.
In here, we are using the singularity container method. To setup the NSCL DAQ, please check NSCL DAQ. After the setup, we assume the file structure is
- /usr/opt/nscl-buster.img
- /usr/opt/opt-buster/
Load the PLX 9054 driver
In order to let the singularity recognize the PLX driver. We have to UNLOAD the Broadcom Driver v8.0 from above, and LOAD the driver in /usr/opt/opt-buster/plx.
To unload the 9054 Driver
/usr/opt/PlxSDK/Bin>sudo ./Plx_unload 9054
Compilation of the driver in the singularity shell
export env variable
Singularity nscl-buster.img:/usr/opt/plx>export PLX_SDK_DIR=$(pwd)
create Driver-<kernal> directory
Singularity nscl-buster.img:/usr/opt/plx>./mkdrivertree
compile the driver
Singularity nscl-buster.img:/usr/opt/plx/Driver-4.19.0-20-amd64>./builddriver 9054
Should able to see something like this:
Build: Plx9054 - PLA: Linux ver ?? - KER: ver 4.19.0-20-amd64 - INC: /lib/modules/4.19.0-20-amd64/build/include - CPU: x86_64 (64-bit Little Endian) - CMP: gcc - TYP: Driver - PLX: 9054 - CFG: Release make[1]: Entering directory '/usr/src/linux-headers-4.19.0-20-amd64' CC [M] /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/ApiFunc.o CC [M] /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Dispatch.o CC [M] /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Driver.o CC [M] /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Eep_9000.o CC [M] /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/ModuleVersion.o CC [M] /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/PciFunc.o CC [M] /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/SuppFunc.o CC [M] /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Chip/9054/PlxChipApi.o CC [M] /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Chip/9054/PlxChipFn.o CC [M] /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Chip/9054/PlxInterrupt.o LD [M] /usr/opt/plx/Driver-4.19.0-20-amd64/Plx9054.o Building modules, stage 2. MODPOST 1 modules CC /usr/opt/plx/Driver-4.19.0-20-amd64/Plx9054.mod.o LD [M] /usr/opt/plx/Driver-4.19.0-20-amd64/Plx9054.ko make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-20-amd64' Driver "Source.Plx9000/Output/Plx9054.ko" built sucessfully
Load the driver in the host system
In the host system, due to the file structure, we need to create a symbolic link
/usr/opt>ln -s opt-buster/plx
go to the plx
/usr/opt>cd opt-buster/plx/Bin /usr/opt/opt-buster/plx/Bin>sudo ./Plx_load 9054
Install: Plx9054 Load module......... Ok (Plx9054.ko) Verify load......... Ok Get major number.... Ok (MajorID = 243) Create node path.... Ok (/dev/plx) Create nodes........ Ok (/dev/plx/Plx9054)
To check, we can
~>sudo lspci -vvv
06:0f.0 Bridge: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge (rev 0b)
Subsystem: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 32, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 18
Region 0: Memory at f7100000 (32-bit, non-prefetchable) [size=256]
Region 2: Memory at f6800000 (32-bit, non-prefetchable) [size=8M]
Region 3: Memory at f7000000 (32-bit, non-prefetchable) [size=1M]
Capabilities: [40] Power Management version 1
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] CompactPCI hot-swap <?>
Capabilities: [4c] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
Not readable
Kernel driver in use: Plx9054
With this, in the singularity shell, the PLX9054 driver is OK. To test, we use the NSCOPE to check
NSCOPE
nscope is a GUI program for setting digitizer parameters, developed by Jeromy Tompkins (modified by David Caussyn). The program used the PLX Driver 7.0, Legacy PixieSDK, CERN ROOT 5.
In the /usr/opt/opt-buster/ddas/5.0-004/bin, there is a nscope.
In order to run nscope, we need other setting files
- cfgPixie16.txt <-- setting
- pxisys.ini <-- required by Legacy PixieSDK, no need to change
- XXXX.set <-- parameters setting file
Those files can be copied from /usr/opt/opt-buster/ddas/5.0-004/share/readout/crate_1.
I created /home/ryan/readout to store those files.
cfgPixie16.txt
We have only 1 crate and 1 digitizer at slot 2.
1 #crateID 1 #number of modules 2 #slot for mod 0 /home/ryan/readout/create_1.set
How to set firmware location?
Run NSCOPE
I set a symbolic link
~/readout>ln -s /usr/opt/ddas/5.0-004/bin/nscope
and run
~/readout>./nscope current working directory /home/ryan/readout
A window will pop out
click the Boot, in the terminal, we have
------------------------ Initializing PXI access... System initialized successfully. Found Pixie-16 module #0, Rev=15, S/N=1314, Bits=16, MSPS=250 Booting Pixie-16 module #0 ComFPGAConfigFile: /usr/opt/ddas/firmware/2.1-000/firmware/syspixie16_current_16b250m.bin SPFPGAConfigFile: /usr/opt/ddas/firmware/2.1-000/firmware/fippixie16_current_16b250m.bin DSPCodeFile: /usr/opt/ddas/firmware/2.1-000/dsp/Pixie16_current_16b250m.ldr DSPVarFile: /usr/opt/ddas/firmware/2.1-000/dsp/Pixie16_current_16b250m.var DSPParFile: /home/ryan/readout/crate_1.set -------------------------------------------------------- Start to boot Communication FPGA in module 0 Start to boot signal processing FPGA in module 0 Start to boot DSP in module 0 All modules ok
It correctly recognize the model, and load the corresponding firmware.
used in Clerion2
The DAQ programs used in Clerion2 are developed by Jame M. Allmond and Tim Gray from ORNL.
PixieDAQ
PixieDAQ is a GUI program for completely controlling and taking data from Pixie16 digitizer, developed by Ryan Tang. The program used the Boardcom PlxSDK 8.0, PixieSDK 3.3 and ROOT 6.24.
I assume the Boardcom PlxSDK, PixieSDK 3.3 are located at
| Boardcom PlxSDK 8.0 | : /usr/opt/PlxSdk |
| PixieSDK 3.3 | : /usr/opt/xia/PixieSDK |
The program can be downloaded at https://github.com/goluckyryan/Pixie16_GUI_DAQ
once downloaded the program. Unless the PlxSDK and PixieSDK are in difference location, using make should able to compile the program.
There is only 1 configuration file #Pixie16.conf
To run,
~/pixieDAQ>./pixieDAQ
Welcome to pixie16 DQ
Removing Pixie16Msg.log
======= check PLX PCI 9054 ...
Found PLX PCI 9054 driver.
Found PLX PCI 9054 card.
PLX PCI 9054 card does not detected problem.
======= Loading Configuration file : Pixie16.config
##########################
Number of Module : 1
Slot Map : 2
Module ID : 0
--- configuration files for module-00 (slot-02)
ComFPGA : ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/firmware/syspixie16_revfgeneral_adc250mhz_r33339.bin
SPFPGA : ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/firmware/fippixie16_revfgeneral_16b250m_r36563.bin
DSP Code : ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/dsp/Pixie16DSP_revfgeneral_16b250m_r35921.ldr
DSP Var : ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/dsp/Pixie16DSP_revfgeneral_16b250m_r35921.var
DSP Par : test_ryan.set
======= Booting Pixie16 System ...
Init Ok
Booting module ...
------------ Module-0
Revision : 15
Serial Num : 1314
ADC Bits : 16
ADC sampling rate : 250
# channels : 16
Boot Ok
Pixie16.conf
########################################### ## Pixie16 Configuration File: ## ########################################### ### only for 1 crate for the moment #Slot Number for Each Module #This must start at 2 and proceed sequentially at the moment # slot modID fpgaID S 2 0 25 #S 3 1 25 ########################################################################################################## ## FPGA Files ID ## ## F Y X File, where Y = fpgaID < 100 ## ## X = 0 (sys*.bin), 1 (fip*.bin), 2 (DSP*.ldr), 3 (DSP*.var), 4(*.set) ## ## Use this to save various FPGA files or to use a mixed board system ## ########################################################################################################## #250MHz 16-bit F 25 0 ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/firmware/syspixie16_revfgeneral_adc250mhz_r33339.bin F 25 1 ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/firmware/fippixie16_revfgeneral_16b250m_r36563.bin F 25 2 ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/dsp/Pixie16DSP_revfgeneral_16b250m_r35921.ldr F 25 3 ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/dsp/Pixie16DSP_revfgeneral_16b250m_r35921.var F 25 4 test_ryan.set
######################################################################################################################## #120 space buffer limit ########################################################################################################################