Pixie16 digitizer: Difference between revisions

From FSU Fox's Lab Wiki
Jump to navigation Jump to search
 
(62 intermediate revisions by the same user not shown)
Line 39: Line 39:
Each model has 16 channels. <span style="color:red">Using connector ?? </red>
Each model has 16 channels. <span style="color:red">Using connector ?? </red>


= Pixie16 firmware =
The firmware can be downloaded from http://download.xia.com/#products/pixie16/firmware/


= Data Structure =
= Data Structure =
Line 44: Line 47:
The recorded data is stored in the extern FIFO (first-in first-out) memory or '''ExtFIFO''' on the digitizer. A block of Data is formed by '''word''', each word is 32 bit long. A minimum data block contains 4 words. When extra information is switched on, they will be stored afterward. The total length of a data block is called '''event length'''.  
The recorded data is stored in the extern FIFO (first-in first-out) memory or '''ExtFIFO''' on the digitizer. A block of Data is formed by '''word''', each word is 32 bit long. A minimum data block contains 4 words. When extra information is switched on, they will be stored afterward. The total length of a data block is called '''event length'''.  


The data structure shown below and taken from the [https://xia.com/wp-content/uploads/2018/04/Pixie16_UserManual.pdf XIA Pixie-16 User Manual version 3.06], page 72.
The data structure shown below and taken from the [https://xia.com/wp-content/uploads/2018/04/Pixie16_UserManual.pdf XIA Pixie-16 User Manual version 3.06] (a copy of the [[:File:Pixie16 UserManual.pdf |Manual]]), page 72.


[[File:Pixie16DataStructure.png|800px|frameless|none|Data structure of Pixie-16. Taken from XIA Pixie-16 User Manual version 3.06.]]
[[File:Pixie16DataStructure.png|600px|frameless|none|Data structure of Pixie-16. Taken from XIA Pixie-16 User Manual version 3.06.]]


The trace (if enabled) is stored after the header.  
The trace (if enabled) is stored after the header.  
The data usually saved in a binary format. The [[Pixie16_digitizer#evtReader_Class|evtReader]] from XIAEventBuilder package can read the binary format in root or in C++.


== Incomplete data block when retrieving ==
== Incomplete data block when retrieving ==
Line 55: Line 60:


If the data is saved continuously, that will be not a problem.
If the data is saved continuously, that will be not a problem.
== Reading data ==
Timothy Gray developed code https://github.com/belmakier/libpixie.
=== XIAEventReader ===
XIA Event Reader is a command line-based data analysis program developed by Ryan Tang from FSU.
The program can be downloaded from https://github.com/goluckyryan/XIAEventBuilder
==== evtReader Class ====
A very useful code is the [https://github.com/goluckyryan/XIAEventBuilder/blob/master/armory/evtReader.h evtReader.h]
This define the evtReader Class and can be loaded with root.
<div class="toccolours mw-collapsible mw-collapsed">
<div style="line-height:1.6;">Example</div>
<div class="mw-collapsible-content">
~>root -e '.L evtReader.h'
root$<span style="color:blue">evtReader</span> * evt = new <span style="color:blue">evtReader</span>("data_file")
root$evt->ReadBlock();evt->data->Print()
root$evt->ScanNumberOfBlock()
</div></div>


= Broadcom PlxSDK =
= Broadcom PlxSDK =


The chip was a product of [https://en.wikipedia.org/wiki/PLX_Technology PLX Technology], acquired by Broadcom Inc in 2014. The chipset driver is now called the Broadcom PCI/PCIe Software Development Kits [https://www.broadcom.com/products/pcie-switches-bridges/software-dev-kits]. The package provides complete documentation and driver. The last release is on 2020.  
The chip was a product of [https://en.wikipedia.org/wiki/PLX_Technology PLX Technology], acquired by Broadcom Inc in 2014. The chipset driver is now called the [https://www.broadcom.com/products/pcie-switches-bridges/software-dev-kits Broadcom PCI/PCIe Software Development Kits ]. The package provides complete documentation and driver. The last release is on 2020.  


Because of that, it only support Linux kernel around that time. I tested on  
Because of that, it only support Linux kernel around that time. I tested on  
Line 74: Line 102:
== Installation ==
== Installation ==


After downloaded the package (here is a [[File:copy]]), unzip it, and we have
After downloaded the package (here is a [[:File:Broadcom PCI PCIe SDK Linux v8 23 Final 2020-11-18.zip| backup copy]]), unzip it, and we have


<div class="toccolours mw-collapsible mw-collapsed" style="width:400px; overflow:auto;">
<div class="toccolours mw-collapsible mw-collapsed" style="width:400px; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;"> File lists </div>
<div style="line-height:1.6;"> File lists </div>
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
  ├── Documentation
  ├── Documentation
Line 106: Line 134:
   Create nodes........ Ok (/dev/plx/Plx9054)
   Create nodes........ Ok (/dev/plx/Plx9054)


In the last step, it loads the 9054 driver to the Linux Kernel. Fro more detail, see [https://fsunuc.physics.fsu.edu/elog/NSCLDAQ/18 | elog]
In the last step, it loads the 9054 driver to the Linux Kernel. Fro more detail, see [https://fsunuc.physics.fsu.edu/elog/Pixie_NSCLDAQ/18 | elog]
 
We can use '''lsmod''' to check the driver is loaded.
 
~>lsmod | grep "Plx9054"


=== Load 9054 driver on start up ===
=== Load 9054 driver on start up ===
To make the driver get loaded on start up, create a file at '''/etc/systemd/system/''', say broadcom.service
To make the driver get loaded on start up, create a file at '''/etc/systemd/system/''', say broadcom.service


<div class="toccolours mw-collapsible mw-collapsed">
<div style="line-height:1.6;">File content</div>
<div class="mw-collapsible-content">
  [Unit]
  [Unit]
  Description=Broadcom PCI/PCIe 9054 Driver
  Description=Broadcom PCI/PCIe 9054 Driver
Line 124: Line 159:
  [Install]
  [Install]
  WantedBy=multi-user.target
  WantedBy=multi-user.target
 
</div></div>
and then
and then


Line 133: Line 168:


'''PixieSDK''' provides all methods to control the digitizer, from parameters setting, writing/reading control register, starting and stopping the DAQ, to retrieving recorded data from its Extern FIFO (first-in-first-out) memory.
'''PixieSDK''' provides all methods to control the digitizer, from parameters setting, writing/reading control register, starting and stopping the DAQ, to retrieving recorded data from its Extern FIFO (first-in-first-out) memory.
== PixieSDK 3.2 ==


The SDK can be downloaded from GitHub in [https://github.com/xiallc/pixie_sdk here].
The SDK can be downloaded from GitHub in [https://github.com/xiallc/pixie_sdk here].
Line 141: Line 174:


[https://docs.pixie16.xia.com/namespacexia_1_1pixie_1_1error.html Pixie API error code]
[https://docs.pixie16.xia.com/namespacexia_1_1pixie_1_1error.html Pixie API error code]
== PixieSDK 3.2 ==


== PixieSDK 3.3 ==
== PixieSDK 3.3 ==


PixieSDK 3.3 is similar to 3.2.
PixieSDK 3.3 is similar to 3.2.
A copy of the programmer's manual is
[[:File:Pixie16 ProgrammerManual.pdf|here]]


== Legacy PixieSDK ==
== Legacy PixieSDK ==
Line 227: Line 265:
  /usr/opt>cd opt-buster/plx/Bin
  /usr/opt>cd opt-buster/plx/Bin
  /usr/opt/opt-buster/plx/Bin>sudo ./Plx_load 9054
  /usr/opt/opt-buster/plx/Bin>sudo ./Plx_load 9054
 
  Install: Plx9054
  Install: Plx9054
   Load module......... Ok (Plx9054.ko)
   Load module......... Ok (Plx9054.ko)
Line 239: Line 277:
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
  ~>sudo lspci -vvv
  ~>sudo lspci -vvv
 
  06:0f.0 Bridge: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge (rev 0b)
  06:0f.0 Bridge: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge (rev 0b)
  Subsystem: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge
  Subsystem: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge
Line 294: Line 332:


===== Run NSCOPE =====
===== Run NSCOPE =====
[[File:Nscope entrance screen.png|270px|thumb|right|The initial window of NSCOPE]]
run in Singularity shell


I set the following symbolic links
  Singularity:~/readout>/usr/opt/ddas/5.0-004/bin/nscope
 
/usr/opt/>ln -s /usr/opt/ddas
  ~/readout>ln -s /usr/opt/ddas/5.0-004/bin/nscope
 
and run in Singularity shell
 
Singularity:~/readout>./nscope
  current working directory /home/ryan/readout
  current working directory /home/ryan/readout


A window will pop out
A window will pop out.
 
[[File:Nscope entrance screen.png|500px|frameless]]


click the <b>Boot</b>, in the terminal, we have
click the <b>Boot</b>, in the terminal, we have
Line 354: Line 385:
The default name of a ring is the user name (so ryan). There is no process as '''producer''' and '''client''', as no process is producing data (producer), and no process is getting the data from the ring buffer (client).
The default name of a ring is the user name (so ryan). There is no process as '''producer''' and '''client''', as no process is producing data (producer), and no process is getting the data from the ring buffer (client).


==== producer:DDASReadout ====
==== producer : DDASReadout ====
To run a producer, in a new terminal, we run another singularity shell
To run a (data) producer or the DDASReadout, in a new terminal, we run another singularity shell
 
  Singularity-01:~/readout>/usr/opt/daq/11.4-013/bin/DDASReadout
  Singularity-01:~/readout>/usr/opt/daq/11.4-013/bin/DDASReadout
<div class="toccolours mw-collapsible mw-collapsed">
<div style="line-height:1.6;">Output</div>
<div class="mw-collapsible-content">
  The new event buffer size will be: 16934
  The new event buffer size will be: 16934
  Using a FIFO threshold of 10240 words
  Using a FIFO threshold of 10240 words
Line 388: Line 421:


It will leave us in the prompt.
It will leave us in the prompt.
</div></div>


==== client:dumper ====
==== client : dumper ====
We also need a client to get the data from the ring buffer.
We also need a (data) client to get the data from the ring buffer.  


In a new terminal, open another Singularity shell
In a new terminal, open another Singularity shell, run the [https://docs.nscl.msu.edu/daq/newsite/nscldaq-11.4/r14117.html dumper] program.


  Singularity-02:~/readout>/usr/opt/daq/11.4-013/bin/dumper --count=50
  Singularity-02:~/readout>/usr/opt/daq/11.4-013/bin/dumper --count=50
Line 406: Line 440:
  +----+------------+-------+-------------+--------+---------+---------+------+-------------+
  +----+------------+-------+-------------+--------+---------+---------+------+-------------+


We can see that the producer and client are there, and we are ready to take data.
We can see that the producer (ID 482) and client (ID 584) are there, and we are ready to take data.


==== Start and Stop ====
==== Start and Stop ====


At the 2nd terminal, the DDASReadout prompt, to start data taking
At the '''DDASReadout''' terminal, there are 3 commands:
%begin
We can see that the 3rd terminal (where the dumper is) will show a lot binary data stream.


to stop, in the 2nd termimal,
{|class='wikitable
%end
!  !! command !!
 
|-
to exit the DDASReadout
|to start data taking ||begin || the '''dumper''' terminal will show a lot of binary data stream.
%exit
|-
|to stop data taking || end  
|-
|to exit the DDASReadout|| exit
|}


==== the data ====
==== the data ====


The dumper program will store data to lmdata_mod0.bin, which is Pixie format.
The dumper program will store data to lmdata_mod0.bin, which is [[Pixie16_digitizer#Data_Structure|Pixie data format]].


=== ReadOutShell ===
=== ReadOutShell ===
 
[[File:ReadoutShell Window.png|270px|thumb|right|Window of the ReadoutShell]]
We are still using the Singularity-container method in here. The files are located in
We are still using the Singularity-container method in here. The files are located in


Line 436: Line 472:
The ReadoutShell is located at  
The ReadoutShell is located at  


  Singularity>/usr/opt/daq/11.4-013/bin/DDASReadout
  Singularity>/usr/opt/daq/11.4-013/bin/ReadoutShell
 
The ReadOutShell will load the ReadOutGUI. See the right picture.
 
===== SSH Pipe for Data Source =====
[[File:DateProvider.png|600px|thumb|right|Screenshot of Data Provider from '''Data source >> List ''' ]]
On the manual bar of the ReadoutGUI, '''Data Source >> Add..''', Select SSHPipe.
{|class='wikitable'
| Host name :|| localhost
|-
| Readout program : || /usr/opt/nscldaq/11.4-013/bin/DDASReadout
|-
|Working directory : || Click the Same as Readout
|-
|Command line options : || --ring=ryan --sourceid = 0
|}


The ReadOutShell will load the ReadOutGUI.
Since we are using '''ryan''' ring. See [[Pixie16_digitizer#Taking_Data hee]].  


== used in Clerion2 ==  
To check, '''Data Source >> List''', you should see the setting.
 
Since we are using ssh to connect the data source, even it is localhost, it will ask you password. So, better use command
~>ssh-copy-id localhost
To copy the ssh key. To create an ssh key if you don't have one, please see [Place holder]
 
A problem for ssh pipe with Singularity container is that, after ssh, it will not run Singularity shell but normal bash shell. To tackle this, two things must be done
#<pre>export export SING_IMAGE=/usr/opt/nscl-buster.img</pre>
#<pre>echo /usr/opt/opt-buster/:/usr/opt > ~/.singularity_bindpoints</pre>
The '''SING_IMAGE''' and '''~/.singularity_bindpoints''' are recognized by the SSHPipe, so that it will go to Singularity constainer.
 
<div class="toccolours mw-collapsible mw-collapsed">
<div style="line-height:1.6;">So, I make a '''ReadOut.sh''' script to simplify to job.</div>
<div class="mw-collapsible-content">
<syntaxhighlight lang="Bash" line>
#!/bin/bash
export VERSION=11.4-013
export USROPT=/usr/opt/opt-buster/
export SINGULARITY_CONTAINER=/usr/opt/nscl-buster.img
export DAQPORTMANAGER=/usr/opt/daq/${VERSION}/bin/DaqPortManager
export DAQPORTMANAGERLOGFILE=$HOME/nscl_daq.log
export DAQPORTMANAGERPIDFILE=$HOME/nscl_daq.pid
export RINGMASTER=/usr/opt/daq/${VERSION}/bin/RingMaster
export RINGMASTERLOGFILE=$HOME/nscl_ring.log
export READOUT=/usr/opt/daq/${VERSION}/bin/ReadoutShell
 
## these two lins are important for ReadOut ssh know to use singularity
export SING_IMAGE=$SINGULARITY_CONTAINER
echo $USROPT:/usr/opt > ~/.singularity_bindpoints
 
singularity exec --shell /bin/bash --bind ${USROPT}:/usr/opt/ ${SINGULARITY_CONTAINER} ${READOUT}
</syntaxhighlight>
</div></div>


A special terminal-based DAQ programs used in '''[[Clerion2]]''' are developed by Jame M. Allmond and Tim Gray from ORNL.
===== Start & Begin =====


== XIAEventReader ==
#'''To boot the Pixie16, or run DDASReadout''', click the '''Start''' button.
#'''To Run''', click the '''Begin''' button


XIA Event Reader is a command line based data analysis programs developed by Ryan Tang from FSU.  
To set the run name, '''Settings >> Event Recording...'''. You can set the '''Run file prefix'''.


The program can be download from https://github.com/goluckyryan/XIAEventBuilder
The data will be stored at the '''Stagearea'''.


=== evtReader Class ===
== used in Clerion2 ==  
A very useful code is the [https://github.com/goluckyryan/XIAEventBuilder/blob/master/armory/evtReader.h evtReader.h]


This define the evtReader Class and can be loaded with root.
A special terminal-based [[Clarion2#DAQ | DAQ]] program used in '''[[Clarion2]]''' are developed by Jame M. Allmond and Tim Gray from ORNL.


== PixieDAQ ==
== PixieDAQ ==
[[File:PixieDAQ entrance.png|270px|thumb|right|initial window of PixieDAQ]]


PixieDAQ is a GUI program for completely controlling and taking data from Pixie16 digitizer, developed by Ryan Tang. The program used the [[#Broadcom PlxSDK | Boardcom PlxSDK 8.0]], PixieSDK 3.3 and ROOT 6.24.
PixieDAQ is a GUI program for completely controlling and taking data from Pixie16 digitizer, developed by Ryan Tang. The program used the [[#Broadcom PlxSDK | Boardcom PlxSDK 8.0]], PixieSDK 3.3 and ROOT 6.24.
Line 466: Line 550:
|}
|}


The program can be downloaded at https://github.com/goluckyryan/Pixie16_GUI_DAQ. Once downloaded the program. Unless the PlxSDK and PixieSDK are in difference location, using make should able to compile the program.  
The program can be downloaded at https://github.com/goluckyryan/Pixie16_GUI_DAQ. Its target platform is '''Debian 10''', using '''CERN ROOT 6.24.06'''. Once downloaded the program. Unless the PlxSDK and PixieSDK are in different locations, using make should be able to compile the program.  


There is only 1 configuration file <b>[[#Pixie16.conf | Pixie16.conf]]</b>
There is only 1 configuration file <b>[[#Pixie16.conf | Pixie16.conf]]</b>
Line 475: Line 559:


  ~/pixieDAQ>./pixieDAQ
  ~/pixieDAQ>./pixieDAQ
A window will pop-out.
<div class="toccolours mw-collapsible mw-collapsed">
<div style="line-height:1.6;">Terminal Ouput</div>
<div class="mw-collapsible-content">
  Welcome to pixie16 DQ  
  Welcome to pixie16 DQ  
  Removing Pixie16Msg.log  
  Removing Pixie16Msg.log  
Line 502: Line 591:
         # channels : 16  
         # channels : 16  
  Boot Ok
  Boot Ok
</div></div>


[[File:PixieDAQ entrance.png|600px|frameless|none]]
===== Pixie16.conf =====
 
<div class="toccolours mw-collapsible mw-collapsed">
<div class="toccolours mw-collapsible mw-collapsed">
===== Pixie16.conf =====
<div style="line-height:1.6;">File content</div>
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
  ###########################################
  ###########################################
Line 536: Line 625:
  #120 space buffer limit  
  #120 space buffer limit  
  ########################################################################################################################
  ########################################################################################################################
</div><div>
</div></div>
 
= Contact =
: Ryan Tang mailto:rtang@fsu.edu

Latest revision as of 16:44, 16 December 2022

Introduction

Illustration of Pixie16 Crate to DAQ PC connection

In order to control the Pixie16 digitizer, the DAQ computer needs to recognize the digitizer via a PCIe card( for example NI PCI-8366).

The digitizer crate (XIA PXI CompactPCI) has 14-slot. The 1st slot must be the communication card (for example, NI PXI-8366). The remaining 13 slots can be used for digitizer and other modules.

The PCIe card and PXI Board requires no driver. The DAQ computer will treat it as a bridge.

In the computer (LINUX), using the command lspci to show the PCI connection 

>lspci

as an example, part of the output: 

04:00.0 PCI bridge: Intel Corporation 41210 [Lanai] Serial to Parallel PCI Bridge (A-Segment Bridge) (rev 09)
04:00.2 PCI bridge: Intel Corporation 41210 [Lanai] Serial to Parallel PCI Bridge (B-Segment Bridge) (rev 09)
05:0f.0 Unassigned class [ff00]: National Instruments PXI-8368 
06:05.0 PCI bridge: Texas Instruments PCI2050 PCI-to-PCI Bridge (rev 02)
06:0f.0 Bridge: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge (rev 0b)

In the above output, the NI PXI-8368 board on the crate is detected. Also the PCI9054 communication chip is detected.

The location of the PCI9054 communication chip on Pixie16 digitizer.

The PCI9054 (or PLX9054) chip is located at the corner of the Pixie16 digitizer.

The PCI9054 chip requires a driver. The driver is provided by Broadcom, and the package is called PlxSDK.

After the driver for the PCI9054 chip is done, the DAQ PC can talk to the digitizer. Next, the PixieSDK has to be installed for controlling the digitizer. The PixieSDK provides methods to control the digitizer.

Versions of Pixie16 Digitizer

In FSU, we have 100MHz (sampling rate) models and 250 MHz models.

500 MHz models are also used in Clarion2.

Each model has 16 channels. Using connector ?? </red>

Pixie16 firmware

The firmware can be downloaded from http://download.xia.com/#products/pixie16/firmware/

Data Structure

The recorded data is stored in the extern FIFO (first-in first-out) memory or ExtFIFO on the digitizer. A block of Data is formed by word, each word is 32 bit long. A minimum data block contains 4 words. When extra information is switched on, they will be stored afterward. The total length of a data block is called event length.

The data structure shown below and taken from the XIA Pixie-16 User Manual version 3.06 (a copy of the Manual), page 72.

Data structure of Pixie-16. Taken from XIA Pixie-16 User Manual version 3.06.

The trace (if enabled) is stored after the header.

The data usually saved in a binary format. The evtReader from XIAEventBuilder package can read the binary format in root or in C++.

Incomplete data block when retrieving

It is possible that an incomplete data block may be retrieved at the beginning or at the end of ExtFIFO.

If the data is saved continuously, that will be not a problem.

Reading data

Timothy Gray developed code https://github.com/belmakier/libpixie.

XIAEventReader

XIA Event Reader is a command line-based data analysis program developed by Ryan Tang from FSU.

The program can be downloaded from https://github.com/goluckyryan/XIAEventBuilder

evtReader Class

A very useful code is the evtReader.h

This define the evtReader Class and can be loaded with root.

Example
~>root -e '.L evtReader.h'
root$evtReader * evt = new evtReader("data_file")
root$evt->ReadBlock();evt->data->Print()
root$evt->ScanNumberOfBlock()

Broadcom PlxSDK

The chip was a product of PLX Technology, acquired by Broadcom Inc in 2014. The chipset driver is now called the Broadcom PCI/PCIe Software Development Kits . The package provides complete documentation and driver. The last release is on 2020.

Because of that, it only support Linux kernel around that time. I tested on

OS Status
Debian 10 OK
Debian 11 Error
Ubuntu 20 Error

Installation

After downloaded the package (here is a backup copy), unzip it, and we have

File lists 
├── Documentation
│   ├── PLX API DLL with Visual Basic.htm
│   ├── PLX_LegacyAPI.pdf
│   ├── PlxRdkReferenceGuide.htm
│   ├── PLX_SDK_General_FAQ.pdf
│   ├── PLX_SDK_Linux_Release_Notes.htm
│   ├── PLX_SDK_Release_Notes.htm
│   └── PlxSdkUserManual.pdf
└── PlxSdk.tar

The tarball contains the driver. I extract the tarball into /usr/opt/PlxSdk. To setup the package (make) and also make the driver: 

~>cd /usr/opt/PlxSdk/
PlxSdk>export PLX_SDK_DIR=$(pwd)
PlxSdk>sudo make                   # This shold make everything, including things in Samples
PlxSdk>cd Driver
PlxSdk/Driver>export PLX_CHIP=9054
PlxSdk/Driver>sudo ./builddriver 9054
PlxSdk>cd ../Bin
PlxSdk/Bin>sudo ./Plx_load 9054
Install: Plx9054
  Load module......... Ok (/usr/opt/PlxSdk/Driver/Source.Plx9000/Output)
  Verify load......... Ok
  Get major number.... Ok (MajorID = 243)
  Create node path.... Ok (/dev/plx)
  Create nodes........ Ok (/dev/plx/Plx9054)

In the last step, it loads the 9054 driver to the Linux Kernel. Fro more detail, see | elog

We can use lsmod to check the driver is loaded. 

~>lsmod | grep "Plx9054"

Load 9054 driver on start up

To make the driver get loaded on start up, create a file at /etc/systemd/system/, say broadcom.service

File content
[Unit]
Description=Broadcom PCI/PCIe 9054 Driver
After=network.target

[Service]
Type=oneshot
Environment=PLX_SDK_DIR=/usr/opt/PlxSdk/
ExecStart=/bin/bash /usr/opt/PlxSdk/Bin/Plx_load 9054
ExecStop=/bin/bash /usr/opt/PlxSdk/Bin/Plx_unload 9054
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

and then 

sudo systemctl daemon-reload 
sudo systemctl enable broadcom.service

Pixie SDK

PixieSDK provides all methods to control the digitizer, from parameters setting, writing/reading control register, starting and stopping the DAQ, to retrieving recorded data from its Extern FIFO (first-in-first-out) memory.

The SDK can be downloaded from GitHub in here.

The methods are listed in this web page.

Pixie API error code

PixieSDK 3.2

PixieSDK 3.3

PixieSDK 3.3 is similar to 3.2.

A copy of the programmer's manual is here

Legacy PixieSDK

DAQ Programs

NSCL DAQ (DDAS)

Pixie16 digitizer is supported by NSCL DAQ via the DDAS (digital data acquistion system) package.

In here, we are using the Singularity container method. To setup the NSCL DAQ, please check NSCL DAQ. After the setup, we assume the file structure is

Path function
/usr/opt/nscl-buster.img Singularity container image
/usr/opt/opt-buster/ NSCL DAQ pre-compiled files

Load the PLX 9054 driver

In order to let the singularity recognize the PLX driver. We have to UNLOAD the Broadcom Driver v8.0 from above, and LOAD the driver in /usr/opt/opt-buster/plx.

To unload the 9054 Driver

/usr/opt/PlxSDK/Bin>sudo ./Plx_unload 9054

Compilation of the driver in the singularity shell

export env variable 

Singularity nscl-buster.img:/usr/opt/plx>export PLX_SDK_DIR=$(pwd)

create Driver-<kernal> directory 

Singularity nscl-buster.img:/usr/opt/plx>./mkdrivertree

compile the driver 

Singularity nscl-buster.img:/usr/opt/plx/Driver-4.19.0-20-amd64>./builddriver 9054
Should able to see something like this:
 Build: Plx9054
 
 - PLA: Linux ver ??
 - KER: ver 4.19.0-20-amd64
 - INC: /lib/modules/4.19.0-20-amd64/build/include
 - CPU: x86_64 (64-bit Little Endian)
 - CMP: gcc
 - TYP: Driver
 - PLX: 9054
 - CFG: Release
 
 make[1]: Entering directory '/usr/src/linux-headers-4.19.0-20-amd64'
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/ApiFunc.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Dispatch.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Driver.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Eep_9000.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/ModuleVersion.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/PciFunc.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/SuppFunc.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Chip/9054/PlxChipApi.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Chip/9054/PlxChipFn.o
   CC [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Source.Plx9000/Chip/9054/PlxInterrupt.o
   LD [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Plx9054.o
   Building modules, stage 2.
   MODPOST 1 modules
   CC      /usr/opt/plx/Driver-4.19.0-20-amd64/Plx9054.mod.o
   LD [M]  /usr/opt/plx/Driver-4.19.0-20-amd64/Plx9054.ko
 make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-20-amd64'
 
 Driver "Source.Plx9000/Output/Plx9054.ko" built sucessfully

Load the driver in the host system

In the host system, due to the file structure, we need to create a symbolic link 

/usr/opt>ln -s opt-buster/plx

go to the plx 

/usr/opt>cd opt-buster/plx/Bin
/usr/opt/opt-buster/plx/Bin>sudo ./Plx_load 9054

Install: Plx9054
  Load module......... Ok (Plx9054.ko)
  Verify load......... Ok
  Get major number.... Ok (MajorID = 243)
  Create node path.... Ok (/dev/plx)
  Create nodes........ Ok (/dev/plx/Plx9054)
To check, we can
~>sudo lspci -vvv

06:0f.0 Bridge: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge (rev 0b)
	Subsystem: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus Bridge
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 18
	Region 0: Memory at f7100000 (32-bit, non-prefetchable) [size=256]
	Region 2: Memory at f6800000 (32-bit, non-prefetchable) [size=8M]
	Region 3: Memory at f7000000 (32-bit, non-prefetchable) [size=1M]
	Capabilities: [40] Power Management version 1
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] CompactPCI hot-swap <?>
	Capabilities: [4c] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
		Not readable
	Kernel driver in use: Plx9054

With this, in the singularity shell, the PLX9054 driver is OK. To test, we use the NSCOPE to check

NSCOPE

nscope is a GUI program for setting digitizer parameters, developed by Jeromy Tompkins (modified by David Caussyn). The program used the PLX Driver 7.0, Legacy PixieSDK, CERN ROOT 5.

In the /usr/opt/opt-buster/ddas/5.0-004/bin, there is a nscope.

In order to run nscope, we need other setting files

file function
cfgPixie16.txt setting
pxisys.ini required by Legacy PixieSDK, no need to change
XXXX.set parameters setting file

Those files can be copied from /usr/opt/opt-buster/ddas/5.0-004/share/readout/crate_1.

I created /home/ryan/readout to store those files.

cfgPixie16.txt

We have only 1 crate and 1 digitizer at slot 2. 

1  #crateID
1  #number of modules
2  #slot for mod 0
/home/ryan/readout/create_1.set

How to set firmware location?

Run NSCOPE
The initial window of NSCOPE

run in Singularity shell 

Singularity:~/readout>/usr/opt/ddas/5.0-004/bin/nscope
current working directory /home/ryan/readout

A window will pop out.

click the Boot, in the terminal, we have 

------------------------
Initializing PXI access... 
System initialized successfully. 
Found Pixie-16 module #0, Rev=15, S/N=1314, Bits=16, MSPS=250

Booting Pixie-16 module #0
	ComFPGAConfigFile:  /usr/opt/ddas/firmware/2.1-000/firmware/syspixie16_current_16b250m.bin
	SPFPGAConfigFile:   /usr/opt/ddas/firmware/2.1-000/firmware/fippixie16_current_16b250m.bin
	DSPCodeFile:        /usr/opt/ddas/firmware/2.1-000/dsp/Pixie16_current_16b250m.ldr
	DSPVarFile:         /usr/opt/ddas/firmware/2.1-000/dsp/Pixie16_current_16b250m.var
	DSPParFile:         /home/ryan/readout/crate_1.set
--------------------------------------------------------

Start to boot Communication FPGA in module 0
Start to boot signal processing FPGA in module 0
Start to boot DSP in module 0
All modules ok

It correctly recognize the model, and load the corresponding firmware.

Taking Data

Make sure

  • the DaqPortManager and RingMaster are running.
  • the NSCOPE can run.
  • the parameters is adjusted for trigger (using NSCOPE).

In this section, we need 3 terminals

  1. for checking the ringbuff
  2. for running the DDASReadout
  3. for running the dumper

Check the RingMaster statue 

Singularity-00:~>/usr/opt/daq/11.4-013/bin/ringbuffer status
+----+------------+-------+-------------+--------+---------+---------+------+-------------+
|Name|data-size(k)|free(k)|max_consumers|producer|maxget(k)|minget(k)|client|clientdata(k)|
+----+------------+-------+-------------+--------+---------+---------+------+-------------+
|ryan|8194        |8194   |100          |-1      |0        |0        |-     |-            |
+----+------------+-------+-------------+--------+---------+---------+------+-------------+

The default name of a ring is the user name (so ryan). There is no process as producer and client, as no process is producing data (producer), and no process is getting the data from the ring buffer (client).

producer : DDASReadout

To run a (data) producer or the DDASReadout, in a new terminal, we run another singularity shell 

Singularity-01:~/readout>/usr/opt/daq/11.4-013/bin/DDASReadout
Output
The new event buffer size will be: 16934
Using a FIFO threshold of 10240 words
Trying to initialize pixie16 
Crate number 1: 1 modules, in slots:2 DSPParFile: /home/ryan/readout/crate_1.set
Module event lengths: 4 
------------------------
Initializing PXI access... 
System initialized successfully. 
Found Pixie-16 module #0, Rev=15, S/N=1314, Bits=16, MSPS=250

Booting Pixie-16 module #0 
	ComFPGAConfigFile:  /usr/opt/ddas/firmware/2.1-000/firmware/syspixie16_current_16b250m.bin
	SPFPGAConfigFile:   /usr/opt/ddas/firmware/2.1-000/firmware/fippixie16_current_16b250m.bin
	DSPCodeFile:        /usr/opt/ddas/firmware/2.1-000/dsp/Pixie16_current_16b250m.ldr
	DSPVarFile:         /usr/opt/ddas/firmware/2.1-000/dsp/Pixie16_current_16b250m.var
	DSPParFile:         /home/ryan/readout/crate_1.set
--------------------------------------------------------

Start to boot Communication FPGA in module 0
Start to boot signal processing FPGA in module 0
Start to boot DSP in module 0
All modules ok 
Module #0 : module id word=0xf1000fa, clock calibration=8
Resetting last channel timestamps on module: 0
Reader created
setup scalers for 1 modules 
Scalers know crate ID = 1
%៛   <=== prompt for enter command to control the readout

It will leave us in the prompt.

client : dumper

We also need a (data) client to get the data from the ring buffer.

In a new terminal, open another Singularity shell, run the dumper program. 

Singularity-02:~/readout>/usr/opt/daq/11.4-013/bin/dumper --count=50

If we go to the 1st terminal and check the ringbuffer 

Singularity-00:~> /usr/opt/daq/11.4-013/bin/ringbuffer status
+----+------------+-------+-------------+--------+---------+---------+------+-------------+
|Name|data-size(k)|free(k)|max_consumers|producer|maxget(k)|minget(k)|client|clientdata(k)|
+----+------------+-------+-------------+--------+---------+---------+------+-------------+
|ryan|8194        |8194   |100          |482     |0        |0        |-     |-            |
|-   |-           |-      |-            |-       |-        |-        |584   |0            |
+----+------------+-------+-------------+--------+---------+---------+------+-------------+

We can see that the producer (ID 482) and client (ID 584) are there, and we are ready to take data.

Start and Stop

At the DDASReadout terminal, there are 3 commands:

command
to start data taking begin the dumper terminal will show a lot of binary data stream.
to stop data taking end
to exit the DDASReadout exit

the data

The dumper program will store data to lmdata_mod0.bin, which is Pixie data format.

ReadOutShell

Window of the ReadoutShell

We are still using the Singularity-container method in here. The files are located in

host system Singularity
NSCL DAQ /usr/opt/opt-buster /usr/opt/

The ReadoutShell is located at 

Singularity>/usr/opt/daq/11.4-013/bin/ReadoutShell

The ReadOutShell will load the ReadOutGUI. See the right picture.

SSH Pipe for Data Source
Screenshot of Data Provider from Data source >> List

On the manual bar of the ReadoutGUI, Data Source >> Add.., Select SSHPipe.

Host name : localhost
Readout program : /usr/opt/nscldaq/11.4-013/bin/DDASReadout
Working directory : Click the Same as Readout
Command line options : --ring=ryan --sourceid = 0

Since we are using ryan ring. See Pixie16_digitizer#Taking_Data hee.

To check, Data Source >> List, you should see the setting.

Since we are using ssh to connect the data source, even it is localhost, it will ask you password. So, better use command 

~>ssh-copy-id localhost

To copy the ssh key. To create an ssh key if you don't have one, please see [Place holder]

A problem for ssh pipe with Singularity container is that, after ssh, it will not run Singularity shell but normal bash shell. To tackle this, two things must be done 

  1. export export SING_IMAGE=/usr/opt/nscl-buster.img
  2. echo /usr/opt/opt-buster/:/usr/opt > ~/.singularity_bindpoints

The SING_IMAGE and ~/.singularity_bindpoints are recognized by the SSHPipe, so that it will go to Singularity constainer.

So, I make a ReadOut.sh script to simplify to job.
#!/bin/bash
export VERSION=11.4-013
export USROPT=/usr/opt/opt-buster/
export SINGULARITY_CONTAINER=/usr/opt/nscl-buster.img
export DAQPORTMANAGER=/usr/opt/daq/${VERSION}/bin/DaqPortManager
export DAQPORTMANAGERLOGFILE=$HOME/nscl_daq.log
export DAQPORTMANAGERPIDFILE=$HOME/nscl_daq.pid
export RINGMASTER=/usr/opt/daq/${VERSION}/bin/RingMaster
export RINGMASTERLOGFILE=$HOME/nscl_ring.log
export READOUT=/usr/opt/daq/${VERSION}/bin/ReadoutShell

## these two lins are important for ReadOut ssh know to use singularity 
export SING_IMAGE=$SINGULARITY_CONTAINER
echo $USROPT:/usr/opt > ~/.singularity_bindpoints

singularity exec --shell /bin/bash --bind ${USROPT}:/usr/opt/ ${SINGULARITY_CONTAINER} ${READOUT}
Start & Begin
  1. To boot the Pixie16, or run DDASReadout, click the Start button.
  2. To Run, click the Begin button

To set the run name, Settings >> Event Recording.... You can set the Run file prefix.

The data will be stored at the Stagearea.

used in Clerion2

A special terminal-based DAQ program used in Clarion2 are developed by Jame M. Allmond and Tim Gray from ORNL.

PixieDAQ

initial window of PixieDAQ

PixieDAQ is a GUI program for completely controlling and taking data from Pixie16 digitizer, developed by Ryan Tang. The program used the Boardcom PlxSDK 8.0, PixieSDK 3.3 and ROOT 6.24.

I assume the Boardcom PlxSDK, PixieSDK 3.3 are located at

Boardcom PlxSDK 8.0 : /usr/opt/PlxSdk
PixieSDK 3.3 : /usr/opt/xia/PixieSDK

The program can be downloaded at https://github.com/goluckyryan/Pixie16_GUI_DAQ. Its target platform is Debian 10, using CERN ROOT 6.24.06. Once downloaded the program. Unless the PlxSDK and PixieSDK are in different locations, using make should be able to compile the program.

There is only 1 configuration file Pixie16.conf

PixieDAQ will boot the digitizer when start.

To run, 

~/pixieDAQ>./pixieDAQ

A window will pop-out.

Terminal Ouput
Welcome to pixie16 DQ 
Removing Pixie16Msg.log 
======= check PLX PCI 9054 ...
Found PLX PCI 9054 driver.
Found PLX PCI 9054 card.
PLX PCI 9054 card does not detected problem.
======= Loading Configuration file : Pixie16.config
########################## 
Number of Module : 1 
 Slot Map :  2	
Module ID :  0	
--- configuration files for module-00 (slot-02)
  ComFPGA : ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/firmware/syspixie16_revfgeneral_adc250mhz_r33339.bin 
   SPFPGA : ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/firmware/fippixie16_revfgeneral_16b250m_r36563.bin 
 DSP Code : ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/dsp/Pixie16DSP_revfgeneral_16b250m_r35921.ldr 
 DSP Var  : ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/dsp/Pixie16DSP_revfgeneral_16b250m_r35921.var 
 DSP Par  : test_ryan.set 
======= Booting Pixie16 System ...
Init Ok
Booting module ...
------------ Module-0 
         Revision : 15 
       Serial Num : 1314 
         ADC Bits : 16 
ADC sampling rate : 250 
       # channels : 16 
Boot Ok
Pixie16.conf
File content
###########################################
## Pixie16 Configuration File:           ##
###########################################

### only for 1 crate for the moment

#Slot Number for Each Module
#This must start at 2 and proceed sequentially at the moment 
#  slot modID fpgaID
S   2    0      25
#S   3    1      25

##########################################################################################################
## FPGA Files ID                                                                                        ##
## F Y X File, where Y = fpgaID < 100                                                                   ##
##                   X = 0 (sys*.bin), 1 (fip*.bin), 2 (DSP*.ldr), 3 (DSP*.var),  4(*.set)              ##
## Use this to save various FPGA files or to use a mixed board system                                   ##
##########################################################################################################

#250MHz 16-bit
F       25       0       ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/firmware/syspixie16_revfgeneral_adc250mhz_r33339.bin
F       25       1       ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/firmware/fippixie16_revfgeneral_16b250m_r36563.bin
F       25       2       ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/dsp/Pixie16DSP_revfgeneral_16b250m_r35921.ldr
F       25       3       ./firmware/pixie16_revf_general_16b250m_35921_2017-01-09/dsp/Pixie16DSP_revfgeneral_16b250m_r35921.var
F       25       4       test_ryan.set 
########################################################################################################################
#120 space buffer limit 
########################################################################################################################

Contact

Ryan Tang mailto:rtang@fsu.edu
Retrieved from "https://fsunuc.physics.fsu.edu/wiki/index.php?title=Pixie16_digitizer&oldid=1496"