PCI Debugging 101
Like many companies, Cirrascale has a few key technologies and components that it uses in a variety of it’s products. A lot of what we do involves being able to move data around; be it around the world between datacenters, around racks within a datacenter, or around components within systems. For moving data between components, we frequently make use of PCI Express (“PCIe”), as it’s become a ubiquitous protocol for interconnecting high-speed components. Lately, we’ve had a lot of customer and partner activity surrounding products making use of our 4-slot 16-lane PCIe risers, meaning I’ve had to spend some time working with customers on integrating their components.
This often means digging through the output of lspci(8) to track down what the PCIe topology looks like, and verifying at a high level that nothing horrifically bad is happening. Unfortunately, the manpage for lspci(8) and pcilib(7) talk a lot about options, and not a lot about how to use the output, so many of our partners and customers can’t just RTFM to get the lay of the PCIe land. What usually happens is that we end up taking a stroll through lspci(8) output.
First, since you as a reader don’t actually have a system to follow along with, let’s start with a simplistic example of checking out the integrated LSI SAS2008 controller on a Gigabyte GA-7PESH2 board. The high-level PCI block diagram looks like this:
Our first stop is to build a map of the PCIe system. Fortunately, PCIe is pretty straightforward in that it’s addressing can always be expressed as a finite tree structure, so the “tree” output of lspci(8) is your friend.
# lspci -tvv
⋮ \-[ 0000 : 00 ]-+-00.0 Intel Corporation Xeon E5/Core i7 DMI2 +-01.0-[01]-- +- 01 . 1 -[02]----00.0 LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] ⋮
Traditional PCI address notation is of the form “Domain:Bus:Device.Function”, so our device’s full address is “ 0000 : 00 : 01 . 1 “. We can use lspci(8) again to explore that device, but instead of the tree view we’ll just use verbose output.
# lspci -s 0000:00:01.1 -vvv
00:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b (rev 07) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 00008000-00008fff
Memory behind bridge: df400000-df9fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: Gigabyte Technology Co., Ltd Device 0000
Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
Address: fee00458 Data: 0000
Masking: 00000003 Pending: 00000000
Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L1, Latency L0 unlimited, L1 <16us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #4, PowerLimit 25.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power+ Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
This verbose output shows us that it’s a PCIe root port, owned by the Intel Xeon CPU (which is where PCIe controllers reside in modern x86 systems). Of particular interest is that the link capabilities (“ LnkCap “) of this particular port are PCIe Gen2 (“ Speed 5GT/s “) with 4 PCIe lanes (“ Width x4 “). A few lines down in the output, the link status (“ LnkSta “) shows that the port is operating at it’s fullest capabilities using 4 lanes of Gen2 PCIe.
Since the PCIe root is actually a bridge, it creates a 2nd PCIe bus, and bridges traffic between the Xeon CPU side (“00″) and that new bus (“02″). Referring back to the lspci(8) tree view, we can spot the address of our intended target device.
⋮ \-[ 0000 : 00 ]-+-00.0 Intel Corporation Xeon E5/Core i7 DMI2 +-01.0-[01]--
+-01.1-[ 02 ]---- 00 . 0 LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] ⋮
Just like before, specifying the particular device (this time “ 0000 : 02 : 00 . 0 “) will get us details.
# lspci -s 0000:02:00.0 -vvv
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) Subsystem: LSI Logic / Symbios Logic Device 3080 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 27 Region 0: I/O ports at 8000 [size=256] Region 1: Memory at df900000 (64-bit, non-prefetchable) [size=16K] Region 3: Memory at df880000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at df400000 [disabled] [size=512K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [d0] Vital Product Data Unknown small resource type 00, will not decode more. Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable+ Count=15 Masked- Vector table: BAR=1 offset=00002000 PBA: BAR=1 offset=00003800 Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq+ ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ IOVSta: Migration- Initial VFs: 16, Total VFs: 16, Number of VFs: 16, Function Dependency Link: 00 VF offset: 1, stride: 1, Device ID: 0072 Supported Page Size: 00000553, System Page Size: 00000001 Region 0: Memory at 00000000df8c0000 (64-bit, non-prefetchable) Region 2: Memory at 00000000df480000 (64-bit, non-prefetchable) VF Migration: offset: 00000000, BIR: 0 Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Kernel driver in use: mpt2sas Kernel modules: mpt2sas
As with the bridge device, we can examine the link capabilities (“ LnkCap “) and see that the part is capable of PCIe Gen2 (“ 5GT/s “) and 8 lanes (“ Width x8“). Although the link is capable of “x8″, the link status (“ LnkSta “) is only 4 lanes (“ Width x4 “). This is because the PCIe bridge which we looked at first only had the capability to support 4 lanes, so when the SAS2008 part and the PCIe bridge negotiated their connection, it had to end up at the lowest common denominator: “x4″.
Now that we’ve found our target device on the PCIe bus, what kinds of interesting things can we learn about it? The answer is of course “lots!”, but let’s focus on the address space reservations and the integrity of the PCIe bus. Since most of what the SAS controller does is move data from disks to system memory, it only needs a few pieces of memory.
⋮ Region 0: I/O ports at 8000 [size=256] Region 1: Memory at df900000 (64-bit, non-prefetchable) [size=16K] Region 3: Memory at df880000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at df400000 [disabled] [size=512K] ⋮
The 256 bytes of I/O ports (“ size=256 “) is located in low address space, where real-mode applications can access it (for configuration and really slow I/O), and then there are 16K (“ size=16K “) and 256K (“ size=256K “) blocks of address space that are (presumably) used by chip-specific drivers to access the buffers on the controller and serve as DMA targets. All-in-all, pretty unexciting.
This SAS controller, like most modern PCI devices, support “ Advanced Error Reporting” (“AER”). We can see from the device status registers (“ DevSta “) that the device has experienced some sort of correctable error (“ CorrErr+ “).
Since the error was a correctable error (“ CorrErr “), the interesting part of the AER output is the correctable error status (“ CESta “). None of the bits are set except for the non-fatal error bit (“ NoNFatalErr+ “). By the name of it (it’s an error, but not a fatal one…and it was correctable!), this sounds like nothing to really be concerned about. Checking if the error is masked or not (“ CEMsk “) shows that the device vendor elected to mask that error (“ NonFatalErr+ “), so they didn’t think it was something that should be tricked up the PCIe device chain and handled either. In truth, the PCI-SIG defines Correctable Non Fatal Errors as “Advisory” errors, and note that is should be used as an indication of a software problem, not to be indicative of an issue with the integrity or functionality of the PCIe bus.
As is probably obvious, there’s not usually much call for looking at PCIe devices that are embedded on a commodity motherboard, but the steps to figuring out where a particular device lives on a PCIe bus, and how it is behaving are the same for most any device. A more common case these days, as I mentioned earlier, is helping customers and partners follow these same steps for products like our GB5400.
That product incorporates two of our PCIe Gen3 risers, and is often used to enable GPGPU applications where all of the GPUs are peers. A trimmed down diagram of the PCIe topology looks something like this:
Following the same steps as before, a tree view in lspci(8) near one of the PCIe switches shows:
# lspci -tvv ⋮ | +-10.6 Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 | +-10.7 Intel Corporation Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 | +-13.0 Intel Corporation Xeon E5 v2/Core i7 R2PCIe | +-13.1 Intel Corporation Xeon E5 v2/Core i7 R2PCIe | +-13.4 Intel Corporation Xeon E5 v2/Core i7 QPI Ring Registers | +-13.5 Intel Corporation Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring | +-16.0 Intel Corporation Xeon E5 v2/Core i7 System Address Decoder | +-16.1 Intel Corporation Xeon E5 v2/Core i7 Broadcast Registers | \-16.2 Intel Corporation Xeon E5 v2/Core i7 Broadcast Registers \-[ 0000:00 ]-+-00.0 Intel Corporation Xeon E5 v2/Core i7 DMI2 +- 02 . 0 -[01-06]----00.0-[02-06]--+-00.0-[03]--+-00.0 NVIDIA Corporation GK110 [GeForce GTX 780] | | \-00.1 NVIDIA Corporation GK110 HDMI Audio | +-04.0-[04]-- | +-08.0-[05]--+-00.0 NVIDIA Corporation GK110 [GeForce GTX 780] | | \-00.1 NVIDIA Corporation GK110 HDMI Audio | \-0c.0-[06]-- +-04.0 Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 +-04.1 Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 +-04.2 Intel Corporation Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 ⋮
We can see that GPU 0 and GPU 2 are NVIDIA GTX 780s, while GPU 1 and GPU 3 are absent. Because the PCIe switch acts like a bridge, there are quite a few more PCIe buses to deal with. This time, our PCIe root is device 2 function 0 , and lspci(8) shows us:
# lspci -vvv -s 0000:00:02.0 00:02.0 PCI bridge: Intel Corporation Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=06, sec-latency=0 I/O behind bridge: 00004000-00005fff Memory behind bridge: dc000000-df1fffff ⋮ LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Latency L0 unlimited, L1 <16us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #0, PowerLimit 75.000W; Interlock- NoCompl- ⋮ Capabilities: [148 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC- UnsupReq+ ACSViol+ UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- ⋮
Here the PCIe connection is 16 lanes of Gen 3 (“ LnkSta “, “ 8GT/s “, “ Width x16“), which matches what we’d expect from our block diagram. The AER flags don’t indicate that any correctable or uncorrectable errors have occurred (“ UESta ” and “ CESta “). Looking at the PCIe side of that same link shows a slightly different story.
# lspci -vvv -s 0000:01:0 0.0
01:00.0 PCI bridge: PLX Technology, Inc. Device 8780 (rev aa) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes ⋮ LnkCap: Port #16, Speed 8GT/s, Width x16, ASPM L1, Latency L0 <4us, L1 <4us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- ⋮ Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP+ SDES- TLP+ FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq+ ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr+ BadTLP+ BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- ⋮
Although the current speed and width of the link are the same as on the PCIe root port (whew! It would be really bad news if they weren’t!), from the PCIe switch point of view there are some correctable errors: Receive, transaction layer, and the non-fatal error we saw in the prior example (“ RxErr+ “, “ BadTLP+“, and “ NonFatalErr+ “). All of the errors are masked (“ CEMsk “), so a reasonable course of action would be to verify bandwidth and latency across the bus meet expectations, and then decide if the errors need to be resolved.
Of course, the root port to PCIe switch connection is only relevant with traffic between GPUs and the CPU (or other GPUs on the other PCIe switch). Also important for most of our customers is inter-PCIe riser traffic between GPUs. To explore that, we’d want to look at the PCIe interface of the GPU itself. From our lspci(8) tree output, the first GPU resides as device 0 on bus 3.
# lspci -vvv -s 0000:03:00.0 03:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Gigabyte Technology Co., Ltd Device 3604 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 158 ⋮ LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 <1us, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- ⋮ Capabilities: [420 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC- UnsupReq+ ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr+ BadTLP+ BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- ⋮
The output tells us that GPU 0 is operating at 16 lanes of Gen3 PCIe, and has similar correctable errors to what we saw before: Receive and transaction layer (“ RxErr+ “, “ BadTLP+ “). Since similar errors are occurring on multiple devices using the PCIe switch, one could make sure the PCIe switch is applying appropriate equalization to the links, and that the actual packets sent appear correctly formed. Realistically, however, tracking down masked correctable errors usually only makes sense if it is materially impacting performance of the PCIe devices. (In this particular situation, bandwidth across all links of the PCIe switch was extremely close to the maximum, and we decided the problems were effectively cosmetic).
It’s unfortunate that much of the lspci(8) documentation is lacking examples, but fortunately most of our Cirrascale customers and partners are happy to work with us to ensure the solutions we’re delivering meet their expectations. After going through the steps above, we usually end up with a successful solution that lets Cirrascale offer uniquely capable platforms for GPGPU, MIC, or I/O intensive applications.
Recent Comments