JIRA: SUP-125 Customer: SNPS (Synopsys) — primary contact: Jerry Miller; SNPS IT: Sam Created: 2026-04-22 02:14 Status (JIRA): open · Current state (as of 2026-06-21 16:53 UTC): Martin sent reply with 3 batched asks (PowerShell ftdi_drop_monitor.ps1 CSV, USB-port swap on B12-14, Dell model+BIOS for both PCs). Vacation handoff to Mikael at send-time — Jerry has been notified in the reply PS. Awaiting Jerry's response. Typical US round-trip: 1–7 working days; geography note in memory.

Summary: Two different CB0 boards reproducibly fail tests on the B12-14 station at Sanmina San Jose. Same CB0 boards pass on station B12-15. CB1 also passes on B12-14 — but CB1 has no on-board FTDI, so the CB0-vs-CB1 comparison is not apples-to-apples (verified by reading both schematics 2026-05-29). The failing chip is CB0's on-board FT4232HL. Channels: A=JTAG MPSSE · B=I2C · C=UART (HCI / BSP) · D=GPIO. April 20 + April 30 primary failures were on channel C (HCI/UART). May 28 + June 3-5 runs: channel A (JTAG) is now the dominant first-failure point (Vivado get_hw_targets failing or hw_server crashing). Cleanup cascade always hits channel B (I2C BoaServicePortLibrary) as a downstream symptom. The cleanest differential is C112803 on B12-14 (10/10 fails) vs C112803 on B12-15 (passes consistently) — same hardware, different fixture. 2026-06-16 update: FTDI driver stack ruled out (identical on both PCs incl. CrowdStrike CSDeviceControl.sys shim); Intel xHCI silicon ruled out (same DEV_7AE0 REV_11). Differential narrowed to the Dell motherboard / BIOS / chassis layer (SUBSYS differs: 0D1A1028 on B12-14 vs 0C7B1028 on B12-15 — both Dell, different SKUs).

Geographic constraint: rig is at Sanmina San Jose (USA); E-Sharp is in Sweden. All physical investigation has to be remotely instructed via Jerry / Sanmina. Travel and ship-back are last resort. Batch asks generously — round-trips are 1–7 days. SNPS InfoSec note (Sam, 2026-06-17): CM site networks are isolated; no internet on test PCs, no third-party remote-install. Self-contained tooling preferred (PowerShell native, or MSI / EXE cacheable at \\synplex\install).

STATUS 2026-06-21 16:53 UTC — reply sent to Jerry; vacation handoff to Mikael

Martin sent a tighter version of the AI-drafted reply, batched 3 asks:

1. Dell model + BIOS version for both B12-14 and B12-15 (msinfo32 → System Model + BIOS Version/Date). Caveat: Martin's note says "even though I do not know how fruitful this will be."
2. USB-port swap on B12-14 — move CB0's USB connection to a different port. Open question in the reply ("Did we ever try to change the USB Port?"); per the 2026-05-29 reply Jerry already tried this with no change, so this may be a reconfirmation step rather than new data.
3. Run ftdi_drop_monitor.ps1 (PowerShell rewrite of the pip-blocked Python script) during a failing run → send back drop_log.csv + matching test log.

Vacation handoff at send-time — Martin out, Mikael picks up incoming Jerry replies. Cold-start brief saved as attachments/SUP-125/_HANDOFF_to_Mikael_2026-06-21.md covering: current state, what's ruled out, working rules for SNPS support, file read order, open trace work, two escalation paths if the current batch comes back ambiguous (PC swap experiment; or DUT-side V3P3_FTDI_AWO PDN review requiring SNPS R&D pull-in).

Engineer aside captured in internal notes (do not relay): "I think we are not fishing in the right water, but let us continue on this track as I am out of ideas." Suggests current host-PC-environment thread may be exhausting itself; DUT-side escalation is the next pivot if (1)–(3) come back empty.

Attachments added to repo 2026-06-21:

• attachments/SUP-125/2026-06-21_reply_sent_to_Jerry.md — definitive sent text
• attachments/SUP-125/2026-06-21_reply_to_Jerry_and_Sam.md — full AI draft (longer/more detailed; kept as reference)
• attachments/SUP-125/2026-06-17_email_thread_jerry_sam.md — email thread snapshot
• attachments/SUP-125/2026-06-17_reply_from_Jerry.md — pip-blocker capture
• attachments/SUP-125/2026-06-16_reply_from_Jerry.md — Device Manager data capture
• attachments/SUP-125/ftdi_drop_monitor.ps1 — PowerShell polling script (attached to JIRA reply)
• attachments/SUP-125/_HANDOFF_to_Mikael_2026-06-21.md — vacation handoff brief

Still-pending TODO (Martin's): two PNG screenshots from Jerry's 06-16 reply (USB_Host_Controller_Ids_B12-14.png / …B12-15.png) are in Jerry's email but not yet saved into the attachments/SUP-125/ folder for archive.

STATUS 2026-06-17 — pip blocker + SNPS IT thread

Jerry's 02:00 reply: tried pip install ftd2xx on B12-14 — failed due to restricted internet. Follow-up email: B12-14 has no internet AND his work laptop has no Python, so he can't even pre-download wheels for offline copy. Asked us to either remote-copy the package (offered IP 10.16.92.47) OR consider rolling the polling script into the existing Mem. XCVR debug sequence.

Sam (SNPS IT) joined the thread:

• CM site networks isolated for security → need local installer (MSI / EXE).
• Pointed at https://ftdichip.com/drivers/d2xx-drivers/ as a candidate — that link is the FTDI D2XX kernel driver, NOT the Python ftd2xx PyPI package (already installed on B12-14 per Jerry's 06-16 driver list). Easy confusion since they share the name.
• Suggested keeping a local copy of installable artifacts at \\synplex\install for future cases.
• Suggested Jerry install Python 3.13 via SNPS self-service client; Jerry pushed back (test PC has 3.10, version-mismatch risk).

Our resolution: skip the pip path entirely by rewriting the polling script in PowerShell using native Get-PnpDevice — no install required on any layer. Politely decline the remote-install offer (no SNPS VPN credentials; their security policy almost certainly forbids it). Acknowledge Sam's \\synplex\install idea for future binary-shipment cases. Clarify the D2XX-driver vs Python-package confusion gracefully.

STATUS 2026-06-16 23:18 UTC — Jerry's reply to 06-12 ask 1 (Device Manager)

Two hypothesis classes ruled out:

Intel DEV_7AE0 is the recent-generation PCH-integrated USB 3.20 xHCI (Raptor Lake / Meteor Lake / Lunar Lake era). Dell SUBSYS_xxxx1028 is Dell-internal SKU ID — different motherboard / BIOS / chassis. Where the differential lives within that layer is still open (BIOS USB settings, selective suspend, ASPM, motherboard root-hub port wiring, VBUS regulation quality, PSU quality on 5V rail, chassis-internal USB header routing).

ftdi_drop_monitor.py CSV not received in this reply — blocked by pip-install friction (see 06-17 follow-up).

STATUS 2026-06-12 09:27 UTC — reply sent to Jerry

Martin sent a trimmed reply focused on the two highest-information-yield asks:

1. Device Manager screenshots from both PCs (B12-14 and B12-15):
   • FTDI driver versions on the four USB Serial Converter A/B/C/D entries (Driver Provider, Driver Version, Driver Date, Digital Signer).
   • USB host controller chipset Hardware Ids (VID/PID/SUBSYS).
2. Run ftdi_drop_monitor.py (Python, attached to JIRA + saved under attachments/SUP-125/) during a failing Memory XCVR test on B12-14:
   • Polls ftd2xx.listDevices() every 100 ms; falls back to pyftdi if ftd2xx isn't installable.
   • Per-channel presence (P000000A/B/C/D) logged to CSV with millisecond timestamps.
   • Will distinguish simultaneous chip-level drop (all 4 channels at once) from staggered per-channel drop.
   • Jerry sends back drop_log.csv + the matching test log → we correlate timestamps to pinpoint the chip-drop event.

Grounding thread parked explicitly in the reply ("I think we can rule out the grounding") — matches the 2026-06-05 disconfirmation (5+5 paired runs, no difference).

Held back from this round — still in the alternatives pocket for the next iteration if (1) and (2) don't resolve:

• USBPcap + Wireshark USB-bus trace (drafted but not asked yet; ~100 MB/5 min, needs USBPcap install)
• Tier-2 PC-swap experiment between B12-14 and B12-15 (intrusive on production stations)
• DUT-side V3P3_FTDI_AWO PDN / decoupling review on CB0 schematic (escalate to SNPS R&D only if host-PC characterisation comes back identical between stations)
• Infrared photo of CB0 during failing run (would address A4 local FT4232HL thermal)

Attachments added to repo 2026-06-12:

• attachments/SUP-125/2026-06-12_reply_to_Jerry.md — full draft + internal notes
• attachments/SUP-125/ftdi_drop_monitor.py — the polling script (attached to JIRA reply)

STATUS 2026-06-05 — major update from Jerry's two replies

Quantitative cross-station evidence is now strong:

• C112803 on B12-14: 10/10 fail (Jerry's June 3-5 consecutive runs; first 5 ungrounded, last 5 grounded — no difference observed)
• C112803 on B12-15: passes several times in the last 7 days (Jerry confirmed same board)
• Other Mem XCVR failures on B12-14 in the last 7 days are unrelated patterns (Jerry can compile if needed)

Fixture-side hypotheses now ruled out by Jerry's experiments:

Hypotheses still live:

• A2 USB host PC environment differential — now leading candidate. With everything fixture-side downstream of the host PC ruled out, the remaining delta between B12-14 and B12-15 must be in the host PC, USB host controller chipset, USB hub topology, USB cable to fixture, or AC mains feeding the PC.
• A4 local FT4232HL thermal (chip on CB0 itself runs hot in B12-14's environment) — not directly tested, can't be eliminated.
• B1-B3 DUT-side latent vulnerability — relatively strengthened because cross-station evidence is so strong, but still requires explaining why same board passes on B12-15.
• C mixed — CB0 is marginal AND B12-14's host environment crosses the threshold.

New failure-mode data (June 3-5 runs on C112803):

Step at which failure hits varies between runs, but all converge on the FT4232HL going down:

• Most common: XCVRTEST_Main → Open ServiceUSB JTAG connection → HwTarget: localhost:3121/xilinx_tcf/Xilinx/P000000A → FAILURE! XCVRTEST_Main execution failed: ERROR: [Common 17-39] 'get_hw_targets' failed due to earlier errors. (= channel A / JTAG dies before Vivado hw_server can connect)
• Sometimes: during Configuring uFPGA DDR4 test → step_ddr4_access (= channel C / HCI dropping mid-test, same as April pattern)
• One run got further: Open HW-Target → P000000A → Get FPGAs → Refresh hw device: xcvu13p_2 → FAILURE! XCVRTEST_Main execution failed: ERROR: [Labtools 27-3176] hw_server failed during internal command. (board C112803, 2026-06-03 16:19:48, station SNPX-SSJ-B12-14, marked "Invalid" by harness)

Interpretation: chip drops "somewhere" within the first 5 minutes of test. The April runs were ~70 s in; June runs often fail at the very first JTAG-connect (within seconds). Whichever channel happens to be active at the moment of drop is the first to surface; channel B (I2C) cleanup always follows.

Identifiers / metadata (verified 2026-05-29 against attachments)

Same-station claim: All four attached .log files plus C108543.xml log the same Accordion hostname A000619. Accordions are physical assets bolted into a specific station, so consistent A000619 across runs is the same station. All runs are on B12-14.

Architecture (verified 2026-05-29 — CB0 schematic + ETT00000004 B12 docs)

CB0's on-board FT4232HL (chip ref SH100001768, schematic page 60)

All four channels share the same V3P3_FTDI_AWO supply, the same ground reference, the same USB upstream. A chip-level perturbation (supply glitch, ground bounce, USB suspend/wake, thermal) hits multiple channels.

Two JTAG paths in the test setup

1. Bsp / on-board JTAG path — CB0 FT4232HL channel A → FTDI JTAG level shifters → JTAG switches → FPGA TAPs. Used by Bsp* tests, IBERT, DDR4-via-MIG. This is where the SUP-125 failure manifests.
2. Clock-board JTAG path (Jerry 2026-06-05 confirmed this is what he meant by "JTAG cables on the clock board") — separate cables from XJTAG dongle (or equivalent) to the clock board. Disconnected for all 10 of Jerry's June 3-5 runs and failure persists → ruled out as contributor.

CB1 vs CB0 access (from BOA12 Control board tester.docx)

"CB1 does not have a supervisor or the BOA service port [...] B12 CB1 is accessed through the SB_CON port, while the B12 CB0 is accessed through the BOA service port."

CB0 has its own FT4232HL on-board (the BOA service port); CB1 doesn't, and uses a sideband connector (SB_CON) into the fixture/TA-side FTDI. CB0-vs-CB1 is therefore not an apples-to-apples comparison for FTDI failures.

Conversation history (JIRA, verbatim)

Jerry Miller — 2026-04-22 02:14 (initial)

Attached H200_12F_CB0_MemoryXcvrTest_20260420-155011.log and C108543.xml

CB0 Memory XCVR test which fails for a communication issue through the FTDI (C108543). This board was tested on B12-15 and this test passes. The CB1 also passes this test without issue. The BP comm. cable was replaced and the USB and HALO connections to the BP were checked and are secure, but the test still fails. This board has been left on the test fixture for investigation. Please review the test logs for this unit (attached) and advise further. Thanks.

Martin Johansson — 2026-04-22 15:24

Yes, there seem to be some instability with the FTDI connection. Both JTAG and UART seem to fail from time to time. I would start inspecting the FTDI chip and components around that one on the DUT.

Jerry Miller — 2026-04-23 02:31

Hi Martin. We've tested several boards that passed this test on the other fixture and they all fail the same way. Given your feedback, I tried swapping out the BP board and it still fails the same way. Let me know what else you'd like us to check. Thanks.

Jerry Miller — 2026-04-24 02:24

Hi Martin. R&D helped us determine that the issue is the rFPGA is stuck in reset because of the extraordinary long JTAG cables (unplugging the cables fixed the issue). We will shorten the JTAG cables and run the JTAG test and the Mem. XCVR test to make sure the fixture works.

(Interpretation 2026-05-29: this is a SEPARATE fixture-side issue R&D found and fixed, not the SUP-125 root cause — Mem XCVR failure persisted after cables were shortened.)

Martin Johansson — 2026-04-24 11:13

Hi Jerry, Thanks for the update! Please update us if the test passes with shorter cable so that we can close the ticket. Thanks, Martin

Jerry Miller — 2026-04-28 01:44

Hi Martin, Per and Mikael were running additional tests today and found that the failure persists, but it intermittent. They still believe it is related to the JTAG chain but not necessarily the cables. I will post their additional findings as known. Thanks.

Martin Johansson — 2026-05-06 13:45

Hi Jerry, Any updates to this issue? Thanks, Martin

Jerry Miller — 2026-05-07 15:56

I was able to get all tests except for the original failure, Mem. XCVR Test on CB0. This was with the JTAG cables removed. The other test failures were fixed by swapping out two clock cables, but this failure is intermittent, e.g. C108543, 4/30/25. Attached is a failed log and passing log for two tests run back-to-back, 4:42 and 4:50 p.m. our time.

Jerry Miller — 2026-05-07 16:00

Attached: H200_12F_CB0_External_ClockTest_20260430-165110.log and H200_12F_CB0_External_ClockTest_20260430-164322.log

Jerry Miller — 2026-05-08 04:13

Hi Martin. I was asked if you could review the logs and get back to us no later than Monday, 11 with any recommendations. Thanks.

Martin Johansson — 2026-05-11 16:56

Hi Jerry, Both the first reported error and the two latest logs point to the same communication issue, just hitting at different points in the test sequence. Common signature: BSP FTDI port P000000B (HAPS-200 12F Service Port C) drops off the USB bus, leading to HCI IO_ERROR and FT_DEVICE_NOT_FOUND. The fact that the 4:51 PM rerun passed the same step that failed at 4:43 PM confirms it's intermittent on the BSP path rather than a test issue.

I'd start with the quick host-side checks — try a different USB port and cable on the PC. If the dropouts persist, investigate the path from the FTDI chip on the DUT, out through Back Panel Connector J17, the BP board (which you've already swapped), and the USB cable to the PC.

Thanks, Martin

(Note: the May 11 reply has a port-naming error — "P000000B (HAPS-200 12F Service Port C)" conflates channel B with the BSP. BSP is actually P000000C. Corrected in Martin's May 29 reply.)

Jerry Miller — 2026-05-29 04:56

Hi Martin, I tried replacing the USB cable, swapping USB ports on the computer, using a gold BP and replacing the BP cable, but no change. There was one test that passed with the JTAG cables disconnected, but I've yet to repeat that. Also, the test was able to get further along before failing when I intentionally grounded the BP board (see attached). Do you have any other suggestions I can try?

Attached: H200_12F_CB0_MemoryXcvrTest_20260528-154355-C112803.log

Martin Johansson — 2026-05-29 14:08

Hi Jerry,

Thanks for the update and the C112803 log. Re-reading the full set — interrupted HCI / FTDI — has held up. The April 20 traceback is unambiguous: hci_handle.dev(sfpga_mig).rd("RunStatus") calls down through hci.ser.d2xx.write() to FTDI's FT_Write, which returns IO_ERROR. So that's the HCI/UART transport on CB0's on-board FT4232HL dropping mid-test (after 70+ seconds of successful DDR4 iterations). Same pattern on the April 30 fail log. May 28 surfaces later, at IBERT — could be the JTAG channel dropping, or an FPGA-side issue; that log is less unambiguous.

Channel layout on the FT4232HL on CB0, for shared reference:

Channel	USB serial	Function
A	P000000A	JTAG MPSSE (Vivado hw_server)
B	P000000B	I2C (BoaServicePortLibrary)
C	P000000C	UART → HCI ("Bsp" / "HAPS-200 12F Service Port C")
D	P000000D	GPIO

All four channels share the same V3P3 supply + ground on the chip, so when it drops it tends to take multiple channels with it. The Failed to reopen P000000B FT_DEVICE_NOT_FOUND in every failure log is the I2C channel failing in cleanup after the chip drops — a downstream symptom, not the primary cause.

1. Is the cooling for the rFPGA the same in both test stations, is the fan running for the rFPGA during the test?
2. How was the BP grounded — what exactly is connected from-to (chassis, Earth, shield)?
3. "There was one test that passed with the JTAG cables disconnected," Which JTAG cables are these?

To get some more data I also suggest:

• Back-to-back retry on C112803 (3–5 reruns, no changes). Is C112803 intermittent like C108543, or is the May 28 IBERT failure deterministic on this board?
• Paired runs grounded vs ungrounded: 3–5 grounded back-to-back, then 3–5 ungrounded, same board same day. Does grounding consistently help?
• Rough sample sizes: how many CB0 boards × how many runs each on B12-14 vs B12-15? Helps us see how strong the cross-station pattern actually is.
• Note USB host-controller driver version, and FTDI driver version on both test stations.

Thanks, Martin

Jerry Miller — 2026-06-05 02:55 (reply 1)

Hi Martin, As promised, below are the responses in the email I sent.

"Is the cooling for the rFPGA the same in both test stations, is the fan running for the rFPGA during the test?"

Every time I observed the fan it was on and I kept the lid closed during test. Also, in between several tests I opened the lid and pressed my fingers against the rFPGA heatsink and the seFPGA and the sFPGA. They never felt hot.

"How was the BP grounded — what exactly is connected from-to (chassis, Earth, shield)?"

Initially, when I grounded the BP I just used an alligator clip but noticed that it occasionally fell off. For these subsequent tests I attached a wire underneath a BP screw and clipped the other end to the CB ground.

"'There was one test that passed with the JTAG cables disconnected,' Which JTAG cables are these?"

These are the JTAG cables on the clock board. For the tests I just ran, I disconnected from the clock board for all tests.

"To get some more data I also suggest:

• Back-to-back retry on C112803 (3–5 reruns, no changes). Is C112803 intermittent like C108543, or is the May 28 IBERT failure deterministic on this board?"

C112803 was the only board still available. I ran it 10 times consecutively. The first 5 runs are without the BP grounded and the last 5 are with it grounded. Although, it failed all 10 tests, and I saw no difference in the results between the ungrounded-BP and grounded-BP sets.

"Paired runs grounded vs ungrounded: 3–5 grounded back-to-back, then 3–5 ungrounded, same board same day. Does grounding consistently help?"

No.

"Rough sample sizes: how many CB0 boards × how many runs each on B12-14 vs B12-15? Helps us see how strong the cross-station pattern actually is."

They are not running any CB0's on B12-14 now because of this issue. So, the only test results there in the last 7 days are the tests I ran. However, this same board, C112803, passed several times on B12-15 in the last 7 days. I did look at the other MEM XCVR failures on that fixture in the last 7 days and they all seem to be for something entirely different. I can compile that data for you if you like.

"Note USB host-controller driver version, and FTDI driver version on both test stations."

Could you send me a screenshot of what to look for? I found about 16 USB devices on the PC and wasn't sure which one's belong to the HOST USB. It did look like the FTDI driver on both was issued by Crowdstrike.

Jerry Miller — 2026-06-05 03:17 (reply 2)

Today, I disconnected the BP from the fixture and connected it externally and grounded it to the CB ground just to be safe. However, when I ran the test on C112803 just to check, it fails in the exact same spot, XCVRTEST_Main -> Open ServiceUSB JTAG connection - Just at the point it is attempting to execute the line, "HwTarget: localhost:3121/xilinx_tcf/Xilinx/P000000A" with the error message:

"FAILURE! XCVRTEST_Main execution failed: ERROR: [Common 17-39] 'get_hw_targets' failed due to earlier errors."

Occasionally, the tests failed earlier during the "Configuring uFPGA DDR4 test:" section of the test, while trying to do the "step_ddr4_access" steps, but one time it got as far as Open HW-Target -> HW-Target: localhost:3121/xilinx_tcf/Xilinx/P000000A -> Get FPGAs -> Refresh hw device: xcvu13p_2, with the error message:

FAILURE! XCVRTEST_Main execution failed: ERROR: [Labtools 27-3176] hw_server failed during internal command.

(H200 12F CB0 Memory XCVR Test (Invalid) Failed 6/3/2026 4:19:48 PM SNPX-SSJ-B12-14)

Is it possible to monitor P000000A and see how it behaves at this portion of the script? It shouldn't be too hard as it fails after 5 mins. Thanks.

Martin Johansson — 2026-06-12 09:27 (as sent)

Hi Jerry,

Thanks for the very thorough work on the C112803 runs — 10/10 with the BP-grounding comparison and the BP-disconnect experiment. I think we can rule out the grounding. FTDI driver versions: Right-click Start → Device Manager Expand Universal Serial Bus controllers Find the four USB Serial Converter A / B / C / D entries (these are the four channels of CB0's FT4232HL when the rig is on) Right-click each → Properties → Driver tab → screenshot We need: Driver Provider, Driver Version, Driver Date, Digital Signer Try this on both B12-14 and B12-15.

While you are there we can also compare the stations host controller chipset: USB host controller chipset: Same Device Manager view Near the top of the Universal Serial Bus controllers section, find the host controller entry (typically something like "Intel(R) USB 3.20 eXtensible Host Controller — 1.0 (Microsoft)" or similar) Right-click → Properties → Details tab Property dropdown → choose Hardware Ids Screenshot

About monitoring P000000A. I've written a small script (ftdi_drop_monitor.py attached) that polls FTDI device enumeration every 100 ms during the failing run and logs to CSV: timestamp, channels present, USB serials. When the chip drops, we'll see the exact timestamp and whether all 4 channels go at once (= chip-level event) or whether they drop staggered (= driver-level or per-channel issue). To run: On B12-14, ensure Python 3.x is installed with the ftd2xx package: pip install ftd2xx Open a separate cmd / PowerShell on the test PC Run: python ftdi_drop_monitor.py --out drop_log.csv Once it's polling (you'll see "polling..." messages), start the failing test Let the test run through its failure Hit Ctrl+C on the polling script after the failure Send back drop_log.csv plus the test log from that same run so we can correlate timestamps

Thanks, Martin

(Trimmed vs the full draft in attachments/SUP-125/2026-06-12_reply_to_Jerry.md: omitted the "where we are after 06-05" hypothesis-update prose, the USBPcap+Wireshark backup option, and the parked-items section. The grounding rule-out is implicit via the one-liner. The fallback pyftdi backend detail is also dropped — script handles it transparently.)

Jerry Miller — 2026-06-16 23:18 (Device Manager reply)

Re: FTDI driver versions - Both PC's were the same for all four CB0 channels: [list of 7 driver-stack entries including FTDI 2.12.36.20, CrowdStrike CSDeviceControl.sys 7.29.20164.0, ftdibus.sys 2.12.36.20, ftd2xx.dll 3.2.21.1, etc. — identical on B12-14 and B12-15].

Screen shots of the USB Host Controller Ids: USB_Host_Controller_Ids_B12-14.png and USB_Host_Controller_Ids_B12-15.png attached.

Screenshots showed Intel(R) USB 3.20 eXtensible Host Controller — 1.20 (Microsoft) on both PCs, with identical VEN_8086 DEV_7AE0 REV_11 but differing SUBSYS: 0D1A1028 on B12-14 vs 0C7B1028 on B12-15 (both Dell, vendor 1028, different SKUs).

Full capture in attachments/SUP-125/2026-06-16_reply_from_Jerry.md.

Jerry Miller — 2026-06-17 02:00 (pip blocker)

Re: ftdi_drop_monitor.py script. Python 3.10 is on this PC. I checked the pip package lists to see if ftd2xx package was installed and it isn't. I tried "pip install ftd2xx" but it came up with error message which seem to be related to the restricted internet connection. I will ask Yiming's team if they can install, but if you can do it remotely that would be good. IP address is 10.16.92.47.

Jerry Miller — 2026-06-17 (follow-up: laptop has no Python)

I investigated installing the ftd2xx package on B12-14, but am unable to because B12-14 doesn't have internet access and because our work laptops do not have Python installed to run pip to download and transfer package files. Can you download the package and remotely copy to the B12-14 instead? I have CC'd Sam in case you have any questions regarding Synopsys InfoSec limitations.

Jerry Miller — 2026-06-17 (alternative: roll into Mem. XCVR debug)

It occurred to me that one possibility might be to roll your FTDI script into a "Mem. XCVR Test" debug sequence for CB0. Would that be easier?

Sam (SNPS IT) — 2026-06-17

@Martin Johansson, just a reminder that CM site networks are isolated for security reasons, so we need local installer (MSI, or EXE). Is this adequate? https://ftdichip.com/drivers/d2xx-drivers/ If this is going to be common, we should keep a local copy on \\synplex\install. Also… @Jerry Miller… You can install Python on your SNPS laptop by going to 'self service client' and select Python.

Jerry Miller → Sam — 2026-06-17

Thanks for the info. However, self-service Python 3.13 version's packages are probably incompatible with 3.10 version packages that's on the test computer.

Sam → Jerry — 2026-06-17

It doesn't sound like much of a version change and might be worth a try. There shouldn't be anything preventing you from installing the current Synplex version on your laptop.

Full email-thread capture in attachments/SUP-125/2026-06-17_email_thread_jerry_sam.md + the two 2026-06-17_reply_from_Jerry.md.

Martin Johansson — 2026-06-21 16:53 (as sent — vacation handoff)

Hi Jerry,

Thanks for the Device Manager pulls. What it tells us:

• FTDI driver stack were identical on both PCs.
• Intel USB host controller silicon were the same chip, same revision.
• The SUBSYS_xxxx1028 field differs: SUBSYS_0D1A1028 on B12-14 vs SUBSYS_0C7B1028 on B12-15. Both are Dell — same xHCI silicon, but different motherboards.

We could check these as well, even though I do not know how fruitfull this will be: For each of B12-14 and B12-15: Settings → System → About → "Device specifications" panel. Send the Device name, Processor, and System type lines. For the model: Win + R, type msinfo32, enter. The System Model field (e.g. "Precision 5860 Tower" or similar). For the BIOS: same msinfo32 window — BIOS Version/Date field.

Did we ever try to change the USB Port? If not, move CB0's USB connection from wherever it is now to a different USB port on B12-14.

Regarding the python script, let us try a Power Shell scrip instead, see attached ftdi_drop_monitor.ps1. Open a cmd or PowerShell window separate from the test runner. Run the command powershell -ExecutionPolicy Bypass -File ftdi_drop_monitor.ps1 -OutPath drop_log.csv. You'll see [polling] heartbeat messages. Start the failing Mem. XCVR test. Let the test run through its failure. Hit Ctrl+C on the polling window after the failure to stop the script. Send back drop_log.csv plus the test log from that same run so we can correlate timestamps.

Regards, Martin

PS. I am going on vacation now, so going forward Mikael will handle this case.

Full sent text + AI draft (with longer rationale) in attachments/SUP-125/2026-06-21_reply_sent_to_Jerry.md + 2026-06-21_reply_to_Jerry_and_Sam.md.

Engineering synthesis (revised 2026-06-16)

Unified failure model — chip-level common-mode on CB0's FT4232HL

When the FT4232HL chip drops (USB-bus-level event, supply/ground perturbation, thermal, suspend), it takes whichever channel is actively in use at that moment first; cleanup paths on other channels then fail as cascade.

In every failing run, channel B (BoaServicePortLibrary) FT_DEVICE_NOT_FOUND appears in the cleanup cascade — downstream symptom of the chip dropping, not the primary cause.

Cross-board / cross-station evidence (2026-06-05):

The cross-station differential is now quantitatively strong. Same board (C112803) deterministic-fail on B12-14, deterministic-pass on B12-15.

Confidence calibration (revised 2026-06-16)

What's been tried (and result)

Working hypotheses — revised 2026-06-16

A. Fixture-side trigger upstream of BP (leading candidate after BP path ruled out):

• A2 USB host PC differential — leading, now narrowed by the 06-16 Device Manager data. Same Intel xHCI silicon on both PCs, but different Dell motherboard SKUs. The actual mechanism (BIOS USB settings — selective suspend, xHCI handoff, ASPM; motherboard root-hub port wiring; VBUS regulation quality; PSU 5V rail quality; chassis-internal USB header routing) is still open and is what the 2026-06-21 asks target.
• A4 local FT4232HL thermal — not directly tested. The FT4232HL is a small SSOP-48 package; local heat (e.g., chassis airflow blocked over CB0 specifically) is possible without rFPGA heatsink being hot. Could be probed indirectly with infrared photo of CB0 during failing run.
• A1 chassis ground via BP — ❌ ruled out 2026-06-05.
• A3 build-drift cables (clock board JTAG) — ❌ ruled out 2026-06-05.

B. DUT-side contribution — relatively strengthened (still requires explaining cross-station pass):

• B1: CB0 has marginal FT4232HL decoupling on V3P3_FTDI_AWO near-failure under specific USB-host bus-power loading. B12-14's host PC delivers 5V/ground in a way that pushes CB0 over.
• B2: CB0 batch variance — would need a "good batch" CB0 to test on B12-14 to confirm.
• B3: CB0 chip-level susceptibility to USB host driver behaviour or chipset-specific timing.

C. Mixed (most plausible a priori) — CB0 has a marginal property AND B12-14's host environment crosses the threshold. The fix could need to be on either side, or both.

Responsibility framing (geographic-constrained)

• Investigation order: upstream fixture-side (host PC + USB topology) first — leading candidate now, still cheapest to investigate.
• Geographic constraint: rig is at Sanmina San Jose (USA); we're in Sweden. Travel and ship-back are last resort. Remote-only investigation continues — Jerry does physical work, we interpret.
• DUT-contribution scenarios remain live — if host-PC differential is characterised and explained but doesn't fix the test, escalate to SNPS R&D for DUT-side hardware investigation.
• Tier-2 experiment to consider: PC swap between B12-14 and B12-15 — moves the most-likely differential into isolation. Intrusive on production stations, save until host-PC characterisation confirms a delta worth chasing.

Cross-references

• CB0 schematic (the DUT, failing — has the on-board FT4232HL): attachments/CustomerDocs/SH100005020_R3.0_B12_Control_Board_0_Schematic_V1.pdf — FT4232HL on page 60, FTDI JTAG level shifters + sFPGA JTAG switches + JTAG debug headers on page 62.
• CB1 schematic (attachments/CustomerDocs/DOC-00001153-R1.0_Schematic_B12_Control_Board_1.pdf) — NOT useful for FTDI comparison (no FT4232 / no FTDI on CB1). CB1 accessed via SB_CON.
• B12 station BOM review (ETT00000004 projectId 38) — confirmed: 2 JTAG paths in test setup (XJTAG dongle + Bsp-via-FTDI), CB1 access via SB_CON, Bsp* test list, equipment list with "JTAG ribbon cables SH100003680? 3".
• Test logs (attachments/SUP-125/):
  • H200_12F_CB0_MemoryXcvrTest_20260420-155011.log + C108543.xml — initial failure (board C108543)
  • H200_12F_CB0_External_ClockTest_20260430-164322.log (fail) + …-165110.log (pass) — back-to-back same board
  • H200_12F_CB0_MemoryXcvrTest_20260528-154355-C112803.log — May 28, board C112803, BP grounded (alligator clip), IBERT failure
  • (pending) June 3-5 run logs — Jerry did not attach in the 06-05 reply; ask if needed
• 2026-06-12 reply package:
  • attachments/SUP-125/2026-06-12_reply_to_Jerry.md — full draft + sent text + internal notes
  • attachments/SUP-125/ftdi_drop_monitor.py — Python channel-drop monitor script (attached to JIRA reply; superseded by PowerShell version)
• 2026-06-16/-17 reply package:
  • attachments/SUP-125/2026-06-16_reply_from_Jerry.md — Device Manager data capture
  • attachments/SUP-125/2026-06-17_reply_from_Jerry.md — pip blocker capture
  • attachments/SUP-125/2026-06-17_email_thread_jerry_sam.md — Sam IT thread + Python self-service discussion
• 2026-06-21 reply package (newly added):
  • attachments/SUP-125/2026-06-21_reply_to_Jerry_and_Sam.md — full AI draft
  • attachments/SUP-125/2026-06-21_reply_sent_to_Jerry.md — definitive sent text
  • attachments/SUP-125/ftdi_drop_monitor.ps1 — PowerShell rewrite (no install required; sent to JIRA)
  • attachments/SUP-125/_HANDOFF_to_Mikael_2026-06-21.md — cold-start brief for Mikael
• (Pending TODO): save USB_Host_Controller_Ids_B12-14.png / …B12-15.png from Jerry's 06-16 email into the attachments/SUP-125/ folder.

Outstanding asks — sent 2026-06-21 16:53 UTC, awaiting Jerry

1. Dell model + BIOS version for both B12-14 and B12-15 (msinfo32 → System Model + BIOS Version/Date). Maps the SUBSYS IDs to actual SKUs; opens BIOS-side investigation. Caveat in reply: "I do not know how fruitful this will be."
2. USB-port swap on B12-14 — move CB0's USB connection to a different physical port. (Note: Jerry already tried this 2026-05-29 with no change; the 06-21 reply re-asks because Martin was uncertain. If Jerry comes back with "already tried that", recognise + move to other asks.)
3. Run ftdi_drop_monitor.ps1 (PowerShell rewrite — no install needed; replaces the pip-blocked Python script). Return drop_log.csv + matching test log → we correlate timestamps to pinpoint chip-drop event and distinguish all-channels-at-once (chip-level) from staggered (per-channel/driver).

Round-trip expectation: US working hours overlap is limited; typical reply latency on this thread has been 1–7 working days. Next chase if no reply by 2026-06-26.

Vacation handoff: Martin out from 2026-06-21. Mikael handles incoming. See attachments/SUP-125/_HANDOFF_to_Mikael_2026-06-21.md for cold-start.

Held back from this round (for follow-up if needed)

• USBPcap + Wireshark trace — drafted but not yet asked. Surface if the PowerShell-script CSV is ambiguous. Larger files (~100 MB/5 min), needs USBPcap to be IT-approved on B12-14.
• PC swap experiment between B12-14 and B12-15 — intrusive, save until host-PC characterisation either confirms a delta worth chasing or comes back clean. First-tier escalation if the 06-21 batch comes back empty.
• Powered USB hub between B12-14 and the fixture — cheap test for VBUS-quality / inrush hypothesis if signs from the 06-21 batch suggest it.
• DUT-side V3P3_FTDI_AWO PDN / decoupling review on CB0 schematic — for escalation to SNPS R&D if host-PC investigation comes back clean. Second-tier escalation per Martin's pre-vacation read ("not fishing in the right water").
• Infrared photo of CB0 during failing run — would address local FT4232HL thermal (A4 variant). Save for later.
• Rolling the polling script into a Mem. XCVR debug sequence (Jerry's 06-17 suggestion) — good for ongoing diagnostics, out of scope for this one-time case; revisit if polling proves useful beyond SUP-125.

What we (E-Sharp, from Sweden) actually do

• Interpret returning logs / photos / measurements.
• Cross-check against B12 station BOM (ETT00000004) and CB0 schematic.
• Design the next round of remote experiments based on what comes back.
• Contact FTDI / Xilinx if a chip-vendor question emerges.
• Only after the remote path is genuinely exhausted: propose travel or ship-back.

Convention for this SNPS Support project (ETT00000003 / projectId 37): customer-shared reference material (schematics, datasheets, specifications) that applies across multiple cases lives in attachments/CustomerDocs/. Per-case material continues to live in attachments/PROJ-NNN/ per the §12c convention.

Why a separate folder: customer schematics like the B12 Control Board (CB0/CB1) are referenced across many cases — having one canonical copy avoids duplicating large PDFs into every case folder and keeps the per-case folders focused on case-specific artefacts (logs, screenshots, scope captures).

Filename convention: preserve customer-supplied filenames so each document stays citable by its customer code (e.g. SH100005020 R3.0, DOC-00001153 R1.0).

Confidentiality: customer-confidential. Repo is private on GitHub. Do not redistribute the files outside E-Sharp.

Current inventory (2026-05-29 initial drop, 5 PDFs, ~14 MB total):

• SH100005020_R3.0_B12_Control_Board_0_Schematic_V1.pdf — CB0 schematic (the DUT in SUP-125)
• DOC-00001153-R1.0_Schematic_B12_Control_Board_1.pdf — CB1 schematic (passes the SUP-125 failing test)
• SH100006251_r0.0_BOA_CB_Production_Test_Board_RC2.pdf — Production Test Board
• SH100006286_R0.0_BOA_CB_LB_Test_Board_schematic.pdf — Loopback Test Board
• DOC-00001539-R1.0_Schematic_BOA_CB_CLK_B2B.pdf — Clock B2B

Documented in:

• attachments/CustomerDocs/_README.md — folder-level guide and file inventory.
• This project's CLAUDE.md — Attachments layout section.
• PLEASE_MJ CLAUDE.md §12c — propagated to be the standard for future customer-support ETT projects.

Reference from entries: use project-root-relative path, e.g. attachments/CustomerDocs/SH100005020_R3.0_B12_Control_Board_0_Schematic_V1.pdf.