Unexpected Interrupt Starving on the 2560

Go To Last Post
16 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi 

 

I'm confused by some symptoms that I've got on my Atmega2560 using avr-gcc 4.7.2 so I thought I'd ask the collective expects here for advice.

 

What I see looks as though execution of my SPI interrupt (SPI_STC_vect) appears to prevent service of my uart RX (USART0_RX_vect) 

 - I appreciate that the SPI_STC_vect has a higher priority than that USART0_RX_vect given its address

 - but I would expect any waiting uart interrupt to be serviced when the SPI interrupt returned: which is not what I appear to see

 

On my SPI I have a SPI-Uart bridge with a fifo (SC16IS752 from NXP)
 - this pulls an IRQ line attached to pinchange irpt on the 2560 when its buffer hits a threshold
 - then I pull out the required bytes via code in the SPI_STC_vect

 

Attaching a logic analyser and toggling leds at the relevant parts I can see that my interrupt on the SPI takes between 20-50us to execute (system clocked at 8MHz)
 - after which there is about 20us before it is entered again: during this time I can see the transaction happen on the SPI data lines (SPI clock at 4MHz)
 - depending on how I have configured the chip I could be pulling up to 60bytes here: which would be ~70 interrupts

 

My problem is that during the entire sequence in and out of the SPI interrupt, no interrupts are serviced on the on-chip uart (incoming serial at 9600baud)
 - code at USART0_RX_vect does not execute at all 
 - with a line thrown during the uart RX code I can see that during traffic it is rock solid regular until the SPI transactions start (taking 26us to execute)
 - it does not get triggered again until the SPI handling is completed (all 70 interrupt executions)
 
Halting code during the SPI interrupt (using AVR studio 6.2) I can confirm that my UCSROB = 0xC1
 - showing that my Uart0  receiver is enabled along with the receive complete interrupt enable 

 - thus I would expect that on returning from the SPI interrupt,  that any waiting UART interrupt would execute
 
At present I've worked around this by configuring the bridge chip to request a service after 4 bytes (thus completing in less than one char on the chip uart)
 - but for my peace of mind however I'd rather understand why this is not working as I would have expected 
 
Is there something else I should look at to see if I can't get these interrupts to interleave? 
 - anything in a 2560 register that I might have missed?
 
Thanks for any suggestions,
All the best,
sd

This topic has a solution.
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Crystal ball is fuzzy unless you show us your code!

 

Trim you project down to the min that shows the problem then post it so others can take a look see.

Your bound to get more replies that way.

 

 

JC

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

The spi handling implements a state machine ( to handle the interaction with the NXP chip)

 - each channel of UART on an external bridge has it's own Interrupt Task (bound to different FIFOs, chip select accessors etc)

 - the following is pretty much how its done

 

Defining the SPI context

 

// SPI Interrupt Task 
typedef struct SIT {
    struct SIT* next; // next in list

    uint16 ID;      // resource abritration
    uint8 owner;

    SITState state; // Label for state being executed
    bool registered;

    // counts
    uint16 TXremain; // min(spaceonchip,path->txAvail)
    uint16 RXremain; // min(path_getRxFree(path),RXonChip)
    bool RXleft;

    uint8 chipChannel;

    Path *path;   // path contains pair of fifos
    SPIContext* spi; // accessors for SPI storage

    bool waitByte;   // internals for pin & byte control
    uint8 inByte;
    uint8 outByte;

    bool pullCS;
    bool wasCSpulled;
    uint8 startByte;

    bool reStart;    // thesholds for registration (e.g RAM buffer vs Chip buffer)
    bool holdoffRX;        
    bool holdoffTX;
    uint8 holdoffTXTicks;
    uint8 holdoffWait;
    uint16 restartLevel;

    uint16 ticksSinceRX; // time counters for higher up the stack
    uint16 ticksSinceTX;
} PACKED SIT;

// Define what an IrptState is:
//  - i.e. a pointer to a fn returning a void* and taking a SIT*

typedef void* (*Proxy)(SIT* sit);
typedef Proxy (*IrptState)(SIT* sit);

static volatile SIT* sActiveSIT;

static volatile IrptState currentState;


ISR(SPI_STC_vect) {
    if (sActiveSIT != NULL) {
        uint8 byte = SPSR;         // force read
        sActiveSIT->inByte = SPDR; // always read
        sActiveSIT->outByte = 0;   // default send NULL
        if (sActiveSIT->pullCS) {
            sActiveSIT->spi->deselect(sActiveSIT->spi->owner);
            sActiveSIT->pullCS = FALSE;
            sActiveSIT->wasCSpulled = TRUE;

        }
        
        // initial state set on transfer or kick
        if (sActiveSIT->waitByte){
            // send a null to get the byte we're waiting for
            sActiveSIT->waitByte = FALSE;
            if ((IrptState)currentState != (IrptState)_RX){
                sActiveSIT->pullCS = TRUE;
            }
        } else {
            currentState = (IrptState)currentState((SIT*)sActiveSIT);

        }
        if ((currentState) && (sActiveSIT->wasCSpulled)) {
            sActiveSIT->spi->select(sActiveSIT->spi->owner);
        }

        if (!currentState) {
            _transferControl();
            if (currentState){
                SPDR = sActiveSIT->outByte;
            }
        } else {
            SPDR = sActiveSIT->outByte; // was written if needed
                                        // will cause return to this interrupt
        }
    }
}

// a typical state looks like 
IrptState _RX(SIT* sit){
    IrptState next;

    if (sit->RXremain){
        fifo_put(sit->path->rxFifo, sit->inByte);
        sit->ticksSinceRX = 0;
        sit->outByte = 0;
        next  = (IrptState)_RX;
        sit->RXremain--;
        if (!sit->RXremain) {
            sit->pullCS = TRUE; // pull select on finish
        }
    } else {

        fifo_put(sit->path->rxFifo, sit->inByte);

        if (sit->TXremain){
            sit->outByte = WRITE_HEADER(THR, sit->chipChannel);
            sit->state = kSITTx;
            next = (IrptState)_TX;
        } else {
            sit->state = kSITCompleteSession;
            next = NULL;
        }
    }

    return next;

}

 

the Uart irpt is:

 

ISR(USART0_RX_vect) {
  
    avr_powmgr_wakeup(WKUP_COMMS);

    idle_prevent(kIdleGateUART, TRUE);

    uartSensor_keepAlive(kKeepAliveTickCount);

    uint8 data = UDR0;   // read first to ensure interrupt cleared

  if (!_shouldSwallow()) {   // some external transceivers we cannot prevent an echo from the hardware
        path_receive(_getPath(), data);
    }
  
}

 - i've confirmed that this does not get called despite the SPI being serviced 70times

 

it's enabled via:

void uartSensor_enable(uint8 framing) {

    UCSR0B = 0;        // disable receiver and transmitter    during setup

    framing &= 0x3F; // make sure we remain asynchronous

    UCSR0C = framing;

    UCSR0B = _BV(RXEN0) | _BV(TXEN0);

    // Enable the Rx interrupt
    UCSR0B |= _BV(RXCIE0);

}

 

I've confirmed that during execution of the SPI interrupt that the correct enables are set on the UCSR0B

 

Does that help explain what's going on?

 

all the best

sd 

 

Last Edited: Fri. Sep 26, 2014 - 11:22 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

There's several routines being called from the ISR() there: deselect(), select(), _transferContol() that you haven't shown.

 

If the suggestion here that one ISR() is blocking another it can only really be that the blocker is takign too long and that's really dependent on what it and all the function it may be calling are doing.

 

I'm kind of guessing that select() and deselect() are trivial "wire wigglers" so my suspicion would be that the intriguingly named _tranfserControl() is where any blocking action is actually happening.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks for the comments (sorry for the delay in getting back - been suffering from some bug I picked from the kids)

 

_transferControl is a handoff

 

depending on the board there can be a few of these bridge chips attached 

 - each channel of which can "own" the bus 

 - their interrupt tasks are in a linked list 

 - if a task has nothing left to do (I've emptied the buffer - or I cannot empty it just now as there't nowhere for it ti go) it's removed from the list 

 


static void _transferControl() {
    // check completed
    SIT* next = sActiveSIT->next;
    SPIContext* last = sActiveSIT->spi;
    bool heldOff = sActiveSIT->holdoffRX | sActiveSIT->holdoffTX;
    if ( ( (!sActiveSIT->RXleft) && !(path_getTxWaiting(sActiveSIT->path)))    || (heldOff) ){
        // nothing left to do so deregister
        // unless we've been told to go back round
        if (!sActiveSIT->reStart){
            SIT_deregister((SIT*)sActiveSIT);
        } else {
            sActiveSIT->reStart = FALSE;
        }
    }
    if (next == NULL) {
        next = list_head(sSIT); // will still be NULL if we've just deregistered
    }
    sActiveSIT = next;
    if (sActiveSIT) {
        last->owner = sActiveSIT->owner;
        _startSIT((SIT*)sActiveSIT);
        currentState = (IrptState)_parseRXLevel;
        sActiveSIT->outByte = sActiveSIT->startByte;
        sActiveSIT->waitByte = TRUE;
    } else {
        if (last->enableIrpt) {
            // last to leave turn off the lights
            last->enableIrpt(FALSE);
            avr_spi_release(last, last->owner);
        }
    }
}

 

the last->enableIrpt here is 

void _mainIrptSet(bool on){
	if (on) {
		SPCR |= _BV(SPIE);
	} else {
		SPCR &= ~_BV(SPIE);
	}
}

 this is the only part of this that writes to a control reg on the chip other than the pin wigglers picked up on earlier

  -  those are chip select lines for the targets on the bus

  - they serve as arbitration as well as clocking forwards a state machine on the remote bridge chip (it is a trigger for the last provided byte to be taken up)

 

What looks to be happening is that the SPI transfer happens sufficiently quickly that its completion interrupt "gets in before" the on-chip Uart receive interrupt that I would want to be serviced

 

Thanks again, I'll keep digging

All the best

sd

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

What are the speeds of the incoming SPI interrupts and UART interrupts?

 

If the UART interrupt happens while the SPI routine is still working, it will just wait, then fire as soon as the spi interrupt handler exits. No problem, unless, of course, the SPI interrupt takes too long and a second character comes to the UART. Now you're in a peck of trouble.

 

You're not waiting for anything within this interrupt handler, are you?

274,207,281-1 The largest known Mersenne Prime

Measure twice, cry, go back to the hardware store

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks Torby

 

no "waits" in the code, they are slightly slower than I would like 

 - most of the handling is dereferencing which fifo ( in internal RAM ) I need use for the bytes

 - then moving through the state machine required to drive these NXP bridges

 

on speeds

 - the spi irpt takes 20-50us and retriggers 18us after being released (during which time the spi transaction happens)

 - UART irpt takes 26us to execute and happens every 1ms (at 9600baud) - unless the SPI is running at the same time (hence this mystery :-( )

 

Perhaps a pic would help.. here's a trace (using some LED lines as indicators in to and out of the irpts)

Trace of UART vs SPI irpts

 

the transaction shown is triggered by the IRQ line going low (on a pin change irpt) and only pulls 4 bytes from the SPI bridge chip (which is attached to a GPS on this board)

 - ideally I'd have that pull 60 bytes at a time to minimise the overhead of setting up the transaction

 - but as you can see it's pushing the execution  of the UART interrupt routine out

 

The confusing thing is that  breaking during the SPI irpt I can see that the UART interrupt is enabled so I would expect the behaviour you describe (where the UART irpt is serviced when the SPI interrupt completes)

 - frustratingly if there's something I've not got set correctly in the chip it doesn't seem to effect operation of either interrupt service except (unless they are operating at the same time )

 

Thanks again

sd

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hmm. 50uS isn't very long to a computer this small. How fast is your cpu clock running?

 

So your SPI interrupt hits, the ISR executes and exits, there's only 17uS left to handle a uart interrupt. When a uart interrupt hits, how long does it take for that ISR to execute?

274,207,281-1 The largest known Mersenne Prime

Measure twice, cry, go back to the hardware store

Last Edited: Fri. Oct 3, 2014 - 01:30 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I'm guessing that your SPI ISR is running too long, and there's another pending SPI interrupt by the time you reach the end of the ISR.  Remember, your SPI ISR started before the LED goes on, and continues after the LED goes off.  The ISR run time is longer than you are measuring.

 

To test this, make a dummy SPI ISR that does nothing other than clear any SPI hardware flags, if needed, and stuffs a dummy byte into the SPI to cause a transfer.  Nothing else.  Then see if your UART interrupt gets recognized.

 

Maybe I missed it, but how do you get 15 SPI interrupts from 4 bytes of SPI data?

 

EDIT: I'm still confused by the 15 interrupts, but your SPI data is only taking about 2us to transfer (4MHz clock).  That means that 2us after you write to SPDR the hardware will generate another SPI interrupt, and you're probably still in the last SPI interrupt ISR at that point, so when you exit the ISR the hardware sees another pending SPI interrupt and takes that over any pending UART interrupt.

 

For kicks, try slowing the SPI clock way down, so that the SPI ISR finishes before the SPI data is fully sent/received and the SPI interrupt request happens.  In this window your UART interrupt should be detected and handled.

Last Edited: Sat. Oct 4, 2014 - 12:48 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks for the input.

 

The led flags I've thrown have been the first and last lines of the interrupt vectors 

CPU is being clocked at 8MHz

 

The activity on the SPI lines does not happen until after the flag at the end of the IRPT routine has thrown

 - the 2uS of activity on the line is pretty much midway between the flags for the end and start of the next vector call

 - it does sound plausible that this could just be because of the pushing/popping in and out of the vector 

 - perhaps I can try and have a look at the assembly and see where the global irpt flag in SREG gets renabled wrt the rest of the code going on (I believe it would be this that cause the lines to start rather than the write to SPDR in this case?)

 

but how do you get 15 SPI interrupts from 4 bytes of SPI data?

It's overhead involved in driving the NXP chip

There are 5 distinct exchanges:  

 - reading & clearing a couple of status registers(around its IRQ handling), reading the RX buffer level, reading the TXbuffer level and the read

 - the chip select throws around each (CS line on trace), each exchange is at least a send & receive (so min 2 irpts)

 - this level of overhead is fine if I'm pulling 60bytes - less so if it's 4 :-(

 

 When a uart interrupt hits, how long does it take for that ISR to execute?

 

There's normally ~15us from the end of the stop bit to the flag I've thrown as the first line in the interrupt (IRQ itself takes 26us)

 - when the vector eventually gets called the byte that is serviced is the earliest on on the line (so it has been shifted in and is waiting)

 

For kicks, try slowing the SPI clock way down, so that the SPI ISR finishes before the SPI data is fully sent/received and the SPI interrupt request happens.  In this window your UART interrupt should be detected and handled.

 

^ this is really good idea 

 - I'll try and find a window to do this shortly

 

Thanks for all the suggestions

All the best,

sd

This reply has been marked as the solution. 
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I've looked a bit deeper (it's an interesting problem).  Here is code that writes SPDR, followed by the ISR return code from another project I had laying around (from a timer ISR).  I suspect your ISR pops about this many registers also, plus or minus a few.

 

 126:	8e bd       	out	0x2e, r24	; 46
 128:	02 c0       	rjmp	.+4      	; 0x12e <__vector_20+0x24>
    }
    else
    {
        ........
 12c:	8e bd       	out	0x2e, r24	; 46
    }
}
 1e8:	ff 91       	pop	r31
 1ea:	ef 91       	pop	r30
 1ec:	9f 91       	pop	r25
 1ee:	8f 91       	pop	r24
 1f0:	6f 91       	pop	r22
 1f2:	5f 91       	pop	r21
 1f4:	4f 91       	pop	r20
 1f6:	3f 91       	pop	r19
 1f8:	2f 91       	pop	r18
 1fa:	0f 90       	pop	r0
 1fc:	0b be       	out	0x3b, r0	; 59
 1fe:	0f 90       	pop	r0
 200:	0f be       	out	0x3f, r0	; 63
 202:	0f 90       	pop	r0
 204:	1f 90       	pop	r1
 206:	18 95       	reti

Depending on the path through the if/else, the cycle count could also include the rjmp (2).  Thus the total cycles to the end of the ISR, after the write to SPDR, could be either 33 (no rjmp) or 35.  Add the one instruction that must execute before another interrupt can be recognized, and you get at least 34 or 36 cycles.  In any case, 16 cycles after the write to SPDR, the SPI interrupt will fire again.  So yes, you are almost certainly triggering the SPI interrupt (after 16 cycles) before the current SPI ISR is finished and the system is ready for another interrupt  (on the order of 34 or 36 cycles).

 

If you keep the write to SPDR at the very end of the ISR, and depending on the number of registers popped in the ISR, you may be able to get by with an SPI clock of 2MHz (32 cycles before next SPI interrupt), or you may need to go to 1MHz (64 cycles before next SPI interrupt).  My guess is 1MHz.

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I suspect your ISR pops about this many registers also

Which is bound to happen when you start calling functions from an ISR(). If possible make everything that is "called" static inline then the compiler will be able to see all the code invokved and know what registers are really being used instead of making the pessimistic guess (that it must) that they might all be.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:

I suspect your ISR pops about this many registers also

Which is bound to happen when you start calling functions from an ISR(). If possible make everything that is "called" static inline then the compiler will be able to see all the code invokved and know what registers are really being used instead of making the pessimistic guess (that it must) that they might all be.

Just to clarify, the ISR I took that postamble code from doesn't call any functions, and contains 14 lines of code.  I think it saves so many registers because the project is over multiple files, and the code generator can't see the whole thing when generating the code.  But I agree that any called functions should be inline if possible.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

 I think it saves so many registers because the project is over multiple files, and the code generator can't see the whole thing when generating the code.

Why does that matter if:

the ISR ... doesn't call any functions

If it pushed all those registers and does not make any form of CALL, RCALL or ICALL then those 14 lines must be doing something so convoluted that the code generator really is using all the registers it's pushing. Otherwise it's a buggy compiler.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Just thought I'd come back and confirm that this problem was caused by precisely the condition kk6gm described 

- the transition was completing prior to the code completely unravelling

 

I have slowed my SPI clock down to 0.5MHz and everything works well

 - which has allowed me to use the bridge chips as proper buffers (so my extra SPI transactions (overhead) are only incurred once for 56 bytes instead of 4)

 

Thanks greatly,

sd

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks for the report.  It's good when we can figure these things out.