Pixie16 digitizer

Introduction

Illustration of Pixie16 Crate to DAQ PC connection

In order to control the Pixie16 digitizer, the DAQ computer needs to recognize the digitizer via a PCIe card( for example NI PCI-8366).

The digitizer crate (XIA PXI CompactPCI) has 14-slot. The 1st slot must be the communication card (for example, NI PXI-8366). The remaining 13 slots can be used for digitizer and other modules.

The PCIe card and PXI Board requires no driver. The DAQ computer will treat it as a bridge.

In the computer (LINUX), using the command lspci to show the PCI connection

>lspci

as an example, part of the output:

04:00.0 PCI bridge: Intel Corporation 41210 [Lanai] Serial to Parallel PCI Bridge (A-Segment Bridge) (rev 09)
04:00.2 PCI bridge: Intel Corporation 41210 [Lanai] Serial to Parallel PCI Bridge (B-Segment Bridge) (rev 09)
05:0f.0 Unassigned class [ff00]: National Instruments PXI-8368 
06:05.0 PCI bridge: Texas Instruments PCI2050 PCI-to-PCI Bridge (rev 02)
06:0f.0 Bridge: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge (rev 0b)

In the above output, the NI PXI-8368 board on the crate is detected. Also the PCI9054 communication chip is detected.

The location of the PCI9054 communication chip on Pixie16 digitizer.

The PCI9054 (or PLX9054) chip is located at the corner of the Pixie16 digitizer.

The PCI9054 chip requires a driver. The driver is provided by Broadcom, and the package is called PlxSDK.

After the driver for the PCI9054 chip is done, the DAQ PC can talk to the digitizer. Next, the PixieSDK has to be installed for controlling the digitizer. The PixieSDK provides methods to control the digitizer.

Versions of Pixie16 Digitizer

In FSU, we have 100MHz (sampling rate) models and 250 MHz models.

500 MHz models are also used in Clarion2.

Each model has 16 channels. Using connector ?? </red>

Data Structure

The recorded data is stored in the extern FIFO (first-in first-out) memory or ExtFIFO on the digitizer. A block of Data is formed by word, each word is 32 bit long. A minimum data block contains 4 words. When extra information is switched on, they will be stored afterward. The total length of a data block is called event length.

The data structure shown below and taken from the XIA Pixie-16 User Manual version 3.06, page 72.

The trace (if enabled) is stored after the header.

Incomplete data block when retrieving

It is possible that an incomplete data block may be retrieved at the beginning or at the end of ExtFIFO.

If the data is saved continuously, that will be not a problem.

Broadcom PlxSDK

The chip was a product of PLX Technology, acquired by Broadcom Inc in 2014. The chipset driver is now called the Broadcom PCI/PCIe Software Development Kits [1]. The package provides complete documentation and driver. The last release is on 2020.

Because of that, it only support Linux kernel around that time. I tested on

OS	Status
Debian 10	OK
Debian 11	Error
Ubuntu 20	Error

installation

After downloaded the package, unzip it, and we have

├── Documentation
│   ├── PLX API DLL with Visual Basic.htm
│   ├── PLX_LegacyAPI.pdf
│   ├── PlxRdkReferenceGuide.htm
│   ├── PLX_SDK_General_FAQ.pdf
│   ├── PLX_SDK_Linux_Release_Notes.htm
│   ├── PLX_SDK_Release_Notes.htm
│   └── PlxSdkUserManual.pdf
└── PlxSdk.tar

The tarball contains the driver. I extract the tarball into /usr/opt/PlxSdk. To setup the package (make) and also make the driver:

~>cd /usr/opt/PlxSdk/
PlxSdk>export PLX_SDK_DIR=$(pwd)
PlxSdk>sudo make                   # This shold make everything, including things in Samples
PlxSdk>cd Driver
PlxSdk/Driver>export PLX_CHIP=9054
PlxSdk/Driver>sudo ./builddriver 9054
PlxSdk>cd ../Bin
PlxSdk/Bin>sudo ./Plx_load 9054
Install: Plx9054
  Load module......... Ok (/usr/opt/PlxSdk/Driver/Source.Plx9000/Output)
  Verify load......... Ok
  Get major number.... Ok (MajorID = 243)
  Create node path.... Ok (/dev/plx)
  Create nodes........ Ok (/dev/plx/Plx9054)

In the last step, it loads the 9054 driver to the Linux Kernel. Fro more detail, see | elog

Load 9054 driver on start up

To make the driver get loaded on start up, create a file at /etc/systemd/system/, say broadcom.service

[Unit]
Description=Broadcom PCI/PCIe 9054 Driver
After=network.target

[Service]
Type=oneshot
Environment=PLX_SDK_DIR=/usr/opt/PlxSdk/
ExecStart=/bin/bash /usr/opt/PlxSdk/Bin/Plx_load 9054
ExecStop=/bin/bash /usr/opt/PlxSdk/Bin/Plx_unload 9054
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

and then

sudo systemctl daemon-reload 
sudo systemctl enable broadcom.service

Pixie SDK

PixieSDK provides all methods to control the digitizer, from parameters setting, writing/reading control register, starting and stopping the DAQ, to retrieving recorded data from its Extern FIFO (first-in-first-out) memory.

PixieSDK 3.2

The SDK can be downloaded from GitHub in here.

The methods are listed in this web page.

Pixie API error code

PixieSDK 3.3

PixieSDK 3.3 is similar to 3.2.

DAQ Programs

NSCL DAQ (DDAS)

Pixie16 digitizer is supported by NSCL DAQ via the DDAS (digital data acquistion system) package.

In here, we are using the singularity container method. To setup the NSCL DAQ, please check NSCL DAQ. After the setup, we assume the file structure is

• /usr/opt/nscl-buster.img
• /usr/opt/opt-buster/

Load the PLX 9054 driver

In order to let the singularity recognize the PLX driver. We have to UNLOAD the Broadcom Driver v8.0 from above, and LOAD the driver in /usr/opt/opt-buster/plx.

To unload the 9054 Driver

/usr/opt/PlxSDK/Bin>sudo ./Plx_unload 9054

Compilation of the driver in the singularity shell

export env variable

Singularity nscl-buster.img:/usr/opt/plx>export PLX_SDK_DIR=$(pwd)

create Driver-<kernal> directory

Singularity nscl-buster.img:/usr/opt/plx>./mkdrivertree

compile the driver

Singularity nscl-buster.img:/usr/opt/plx/Driver-4.19.0-20-amd64>./builddriver 9054

Should able to see something like this:

 Build: Plx9054
 
 - PLA: Linux ver ??
 - KER: ver 4.19.0-20-amd64
 - INC: /lib/modules/4.19.0-20-amd64/build/include
 - CPU: x86_64 (64-bit Little Endian)
 - CMP: gcc
 - TYP: Driver
 - PLX: 9054
 - CFG: Release
 
 make[1]: Entering directory '/usr/src/linux-headers-4.19.0-20-amd64'
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/ApiFunc.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Dispatch.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Driver.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Eep_9000.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/ModuleVersion.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/PciFunc.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/SuppFunc.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Chip/9054/PlxChipApi.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Chip/9054/PlxChipFn.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Chip/9054/PlxInterrupt.o
   LD [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Plx9054.o
   Building modules, stage 2.
   MODPOST 1 modules
   CC      /usr/opt/plx/Driver-4.19.0-20-amd64/Plx9054.mod.o
   LD [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Plx9054.ko
 make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-20-amd64'
 
 Driver "Source.Plx9000/Output/Plx9054.ko" built sucessfully
 

Load the driver in the host system

In the host system, due to the file structure, we need to create a symbolic link

/usr/opt>ln -s opt-buster/plx

go to the plx

/usr/opt>cd opt-buster/plx/Bin
/usr/opt/opt-buster/plx/Bin>sudo ./Plx_load 9054

Install: Plx9054
  Load module......... Ok (Plx9054.ko)
  Verify load......... Ok
  Get major number.... Ok (MajorID = 243)
  Create node path.... Ok (/dev/plx)
  Create nodes........ Ok (/dev/plx/Plx9054)

To check, we can

~>sudo lspci -vvv

05:0f.0 Unassigned class [ff00]: National Instruments PXI-8368
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- 

Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

	Interrupt: pin A routed to IRQ 11 
	Region 0: Memory at f7300000 (32-bit, non-prefetchable) [disabled] [size=8K]

06:05.0 PCI bridge: Texas Instruments PCI2050 PCI-to-PCI Bridge (rev 02) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Bus: primary=06, secondary=07, subordinate=07, sec-latency=32
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [dc] Power Management version 1
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

06:0f.0 Bridge: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge (rev 0b)
	Subsystem: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 18
	Region 0: Memory at f7100000 (32-bit, non-prefetchable) [size=256]
	Region 2: Memory at f6800000 (32-bit, non-prefetchable) [size=8M]
	Region 3: Memory at f7000000 (32-bit, non-prefetchable) [size=1M]
	Capabilities: [40] Power Management version 1
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] CompactPCI hot-swap <?>
	Capabilities: [4c] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
		Not readable
	Kernel driver in use: Plx9054

With this, in the singularity shell, the PLX9054 driver is OK. To test, we use the NSCOPE to check

NSCOPE

nscope is a GUI program for setting digitizer parameters, developed by Jeromy Tompkins (modified by David Caussyn). The program used the Legacy PixieSDK, CERN ROOT 5.

In the /usr/opt/opt-buster/ddas/5.0-004/bin, there is a nscope.

In order to run nscope, we need other setting files

• cfgPixie16.txt <-- setting
• pxisys.ini <-- required by pixie, no need to change
• XXXX.set <-- parameters setting file

used in Clerion2

The DAQ programs used in Clerion2 are developed by Jame M. Allmond and Tim Gray from ORNL.

PixieDAQ

PixieDAQ is a GUI program for completely controlling and taking data from Pixie16 digitizer, developed by Ryan Tang. The program used the PixieSDK 3.3 and ROOT 6.24.