SPI Debugging Hell
SPI Debugging Hell
Embedded Files
SPI can't be that hard, right? RIGHT?
SPI can't be that hard, right? RIGHT?
Even Dave knew I was going to be in trouble.
When I swapped from using 7 GPIO pins for CS to a shift register for CS, I thought the process would be straight-forward, just update my code to shift a single LOW through a serial-in, parallel-out shift register. I could pulse that LOW bit through the register after each SPI command set to select the next TC reader in the series, and then reset the shift register at the end to start over again. This concept of CS assertion overlooks a critical aspect of how SPI is implemented on the peripheral side: mainly that one can't send two write commands in a row without possibly encountering unexpected behavior. I'll first explain why that's an issue and how it's supposed to be worked around (spoilers: it's using CS), then I'll walk through my multi-day troubleshooting process including the dead ends.
When I swapped from using 7 GPIO pins for CS to a shift register for CS, I thought the process would be straight-forward, just update my code to shift a single LOW through a serial-in, parallel-out shift register. I could pulse that LOW bit through the register after each SPI command set to select the next TC reader in the series, and then reset the shift register at the end to start over again. This concept of CS assertion overlooks a critical aspect of how SPI is implemented on the peripheral side: mainly that one can't send two write commands in a row without possibly encountering unexpected behavior. I'll first explain why that's an issue and how it's supposed to be worked around (spoilers: it's using CS), then I'll walk through my multi-day troubleshooting process including the dead ends.
Welcome to hell! At the height of the debugging process, I was using the complete TC reader, a multimeter, a 4-channel scope with SPI decoding, and my laptop hooked up to the controller reading out print statements.
The SPI timing diagram for the MAX31856 TC reader I used for this project. Normally, I skip over these when reading a datasheet because it feels almost unnecessary, it's just standard SPI, and the library I'd use to communicate with it inherently handles this level of abstraction. Dear reader, note the explicit de-assertion and re-assertion of the CS line at the end of the command.
Hidden in plain sight on page 15 of the datasheet is the implication that two completely different addresses (commands) cannot be done while CS asserted. Only adjacent, sequential addresses can be accessed.
Why CS? And (Mostly) Unwritten Rules
Why CS? And (Mostly) Unwritten Rules
Put simply, CS is a good way to make it easy to talk to many different (or identical devices) on a data bus quickly and efficiently (especially at the protocol level) at the compromise of increased IO. SPI peripheral chip designers are aware of this, and as a result, some unwritten (or at least assumed) rules are baked into the hardware of most SPI peripherals. As a quick background, almost every single SPI transaction involves the following:
1. The CS to the peripheral is asserted by the controller (in most peripherals I've worked with, this is done by pulling LOW the CS on the peripheral).
Put simply, CS is a good way to make it easy to talk to many different (or identical devices) on a data bus quickly and efficiently (especially at the protocol level) at the compromise of increased IO. SPI peripheral chip designers are aware of this, and as a result, some unwritten (or at least assumed) rules are baked into the hardware of most SPI peripherals. As a quick background, almost every single SPI transaction involves the following:
1. The CS to the peripheral is asserted by the controller (in most peripherals I've worked with, this is done by pulling LOW the CS on the peripheral).
2. A command (read or write) from the controller is shifted into the peripheral. If data needs to be sent back to the controller from the peripheral, the controller will shift extra empty data in (really the peripheral just needs the same number of clock pulses to number of data bits that are sent out; however, this does mean whatever is on the data in line is sent in -- the peripheral is just smart enough to not read it), which in turn pushes out the useful data.
2. A command (read or write) from the controller is shifted into the peripheral. If data needs to be sent back to the controller from the peripheral, the controller will shift extra empty data in (really the peripheral just needs the same number of clock pulses to number of data bits that are sent out; however, this does mean whatever is on the data in line is sent in -- the peripheral is just smart enough to not read it), which in turn pushes out the useful data.
3. The CS is de-asserted by the controller ending the transaction.
3. The CS is de-asserted by the controller ending the transaction.
Note how in step 2, command was very intentionally written as singular. The intentionality of this part of the protocol cannot be overstated. If I was designing an SPI-capable chip, I would design the physical logic so it expects a read/write command on assertion of the CS pin. Any data sent to my chip after the command would either be data in or extra clock pulses to shift data out.
Note how in step 2, command was very intentionally written as singular. The intentionality of this part of the protocol cannot be overstated. If I was designing an SPI-capable chip, I would design the physical logic so it expects a read/write command on assertion of the CS pin. Any data sent to my chip after the command would either be data in or extra clock pulses to shift data out.
If I wanted a user to be able to send a new command without reasserting CS again, I'd either have to specify explicitly in the datasheet (see left) that the chip will automatically move to the next command after the right amount of bits have been shifted through, or, and this is the real crux, I'd have to design my chip so that it would listen for new commands without CS being asserted which would inherently require any possible command not being the same as any possible data that could be shifted in. In many cases, especially when I only want my peripheral chip to only have one (8-bit) byte commands, I could only shift in data that has a maximum value of 256 - the number of commands I want the chip to have, super limiting and I imagine a pain to implement at a physical logic level!
If I wanted a user to be able to send a new command without reasserting CS again, I'd either have to specify explicitly in the datasheet (see left) that the chip will automatically move to the next command after the right amount of bits have been shifted through, or, and this is the real crux, I'd have to design my chip so that it would listen for new commands without CS being asserted which would inherently require any possible command not being the same as any possible data that could be shifted in. In many cases, especially when I only want my peripheral chip to only have one (8-bit) byte commands, I could only shift in data that has a maximum value of 256 - the number of commands I want the chip to have, super limiting and I imagine a pain to implement at a physical logic level!
As a result, de-asserting CS becomes just as important as asserting CS: chip designers need a tidy end to transactions, and that's a LOT easier to implement on an already more complex controller than on a cheap, basic, and, depending on the project, numerous peripheral. This is the mistake I made; I tried to send multiple commands without cycling through the shift register. As a result, a lot of garbage came out that took days to realize was garbage and not something else.
As a result, de-asserting CS becomes just as important as asserting CS: chip designers need a tidy end to transactions, and that's a LOT easier to implement on an already more complex controller than on a cheap, basic, and, depending on the project, numerous peripheral. This is the mistake I made; I tried to send multiple commands without cycling through the shift register. As a result, a lot of garbage came out that took days to realize was garbage and not something else.
My Long, Tedious Debugging Process
My Long, Tedious Debugging Process
Since the only surface-level change to the whole reader from the first version was the addition of the shift register, I began debugging by examining it, and in particular, the large number of potential shorts I had from now having all the CS lines right next to each other.
Since the only surface-level change to the whole reader from the first version was the addition of the shift register, I began debugging by examining it, and in particular, the large number of potential shorts I had from now having all the CS lines right next to each other.
I used a multimeter to check every CS pin and shift register pin for cross-talk. Nothing.
I used a multimeter to check every CS pin and shift register pin for cross-talk. Nothing.
I questioned the reliability of my connections and realized that my first iteration had the potential for cross-talk in the future. I created a new board for the shift register with less exposed connections. Still not working.
I questioned the reliability of my connections and realized that my first iteration had the potential for cross-talk in the future. I created a new board for the shift register with less exposed connections. Still not working.
I looked at the datasheet for the shift register; it has two serial inputs and what appears to be an AND gate for the inputs. The truth table also showed AND gate behavior for the inputs (see left). I soldered the two serial inputs together. Still nothing.
I looked at the datasheet for the shift register; it has two serial inputs and what appears to be an AND gate for the inputs. The truth table also showed AND gate behavior for the inputs (see left). I soldered the two serial inputs together. Still nothing.
I hooked up an oscilloscope to the shift register pins to see if they were going low when they were supposed to. They were. I desoldered the bridge between the two inputs.
I hooked up an oscilloscope to the shift register pins to see if they were going low when they were supposed to. They were. I desoldered the bridge between the two inputs.
I did notice something weird: the first CS pulse was really fast, only 10s of us while the other CS pulses were 20 ms. I was going to have to look at the data on the SPI bus itself.
I did notice something weird: the first CS pulse was really fast, only 10s of us while the other CS pulses were 20 ms. I was going to have to look at the data on the SPI bus itself.
I hooked up the scope to the SPI bus and the first CS and opened up the datasheet and the Arduino code library for the chip. After spending a while staring at hex numbers, I figured out what was being sent. During a read, the controller first polls the chip to make sure the reading was done. For the first chip connected to the shift register, it immediately read the data. For later chips, it waited a long time then gave up. That was interesting.
I hooked up the scope to the SPI bus and the first CS and opened up the datasheet and the Arduino code library for the chip. After spending a while staring at hex numbers, I figured out what was being sent. During a read, the controller first polls the chip to make sure the reading was done. For the first chip connected to the shift register, it immediately read the data. For later chips, it waited a long time then gave up. That was interesting.
Looking into the code further, it became clear that the read mode would be important. There's two read modes for the TC chip, continuous read and "one-shot", where the controller must first command the TC chip to take a reading and then wait for the reading and associated conversions to take place before data can be read out. The conversion, regardless of read mode, takes a long time, ~150-200 ms. I first tried adding a significant delay between the take data and read commands in order to give time for the conversion. This didn't work with either read modes and I ended up switching back to one shot which was the mode I originally had the code working with before the shift register was installed.
Looking into the code further, it became clear that the read mode would be important. There's two read modes for the TC chip, continuous read and "one-shot", where the controller must first command the TC chip to take a reading and then wait for the reading and associated conversions to take place before data can be read out. The conversion, regardless of read mode, takes a long time, ~150-200 ms. I first tried adding a significant delay between the take data and read commands in order to give time for the conversion. This didn't work with either read modes and I ended up switching back to one shot which was the mode I originally had the code working with before the shift register was installed.
Reading up more on the one shot mode, I found another important paragraph in the datasheet: in one shot mode, the conversion only begins when CS is de-asserted. I tried moving the one shot command to a separate loop through the shift register from the read command. In this way, the one shot command would be sent to each TC reader chip with the CS then being de-asserted as the shift register would be cleared then reset for the read. This too didn't work.
Reading up more on the one shot mode, I found another important paragraph in the datasheet: in one shot mode, the conversion only begins when CS is de-asserted. I tried moving the one shot command to a separate loop through the shift register from the read command. In this way, the one shot command would be sent to each TC reader chip with the CS then being de-asserted as the shift register would be cleared then reset for the read. This too didn't work.
At this point, after sleeping on it, I came back the next day and disconnected one of the CS pins on one of the chips from the shift register and plugged it back into the regular Arduino GPIO pins and used the scope to monitor a "normal" transaction using the vanilla library. Glaringly clear was that the CS pin is de-asserted after every single transaction. This was the missing piece. After reworking the code so the shift register would always de-assert CS to each board after any command (either by shifting to the next board or by clearing the register), I finally had a working TC reader using the shift register.
At this point, after sleeping on it, I came back the next day and disconnected one of the CS pins on one of the chips from the shift register and plugged it back into the regular Arduino GPIO pins and used the scope to monitor a "normal" transaction using the vanilla library. Glaringly clear was that the CS pin is de-asserted after every single transaction. This was the missing piece. After reworking the code so the shift register would always de-assert CS to each board after any command (either by shifting to the next board or by clearing the register), I finally had a working TC reader using the shift register.
Although the code is significantly less elegant (the shift register does a WHOLE lot of shifting), the actual latency hasn't increased much since each shift is on the order of 1 us, while the conversion time for the TC chip is 150 ms. Suffice to say that adding the shift register has increased the loop time by less than 1 ms while dramatically reducing the amount of IO used. While debugging took quite some time, I learned a whole lot about shift registers, SPI, and low-level communications as a whole!
Although the code is significantly less elegant (the shift register does a WHOLE lot of shifting), the actual latency hasn't increased much since each shift is on the order of 1 us, while the conversion time for the TC chip is 150 ms. Suffice to say that adding the shift register has increased the loop time by less than 1 ms while dramatically reducing the amount of IO used. While debugging took quite some time, I learned a whole lot about shift registers, SPI, and low-level communications as a whole!
Google Sites
Report abuse