LSU issue

[bantierr]

Hi,

So, I have continued poking around since my last issue, and I found an interesting memory re-ordering scenario that should not be allowed.

Observed behaviour

This is the scenario, which happens at in the ELF file attached.

hart0 instructions:

core   0: 0x80002540 (0x00a5b023) sd      a0, 0(a1)                     -----> stores value 0x8006decc
core   0: 0x80002544 (0x3ee56663) bltu    a0, a4, pc + 1004
core   0: 0x80002548 (0x00058903) lb      s2, 0(a1)                      -----> loads value 0xffffffffffffffcc
core   0: 0x8000254c (0x01294933) xor     s2, s2, s2              
core   0: 0x80002550 (0x0005ca83) lbu     s5, 0(a1)                     -----> loads 0x0 (VIOLATION)
core   0: 0x80002554 (0x020abab3) mulhu   s5, s5, zero
core   0: 0x80002558 (0x81291d63) bne     s2, s2, pc - 4070
core   0: 0x8000255c (0x0088b023) sd      s0, 0(a7) (different address, ignore)
core   0: 0x80002560 (0x0005e103) lwu     sp, 0(a1)                    -----> loads value 0x8006decc

hart1 instructions:

Execute some stores (only 0) and loads to the same address a a1

Expected Behaviour

So, the problem is the following:

1. the store at PC=0x80002540 has performed, because the loads at PC=0x80002548 read its value.
2. The load at address PC=0x80002550 reads 0, which is a stale value
   This violates the RVWMO CoRR

Extras

Command used:

sbt "runMain naxriscv.platform.tilelinkdemo.SocSim --withFloat --withDouble --withRvc --xlen 64 --no-rvls --nax-count 3 --trace --pass-symbol pass"

I hope this is clear and correct. I am unsure of how reproducible this is, so I have attached the trace, elf and log.
Thank you for the help!
Best,
Quentin :)

mem_issue.zip

[Dolu1990]

Hi,
Looking at the wave, it seems related to a memory coherency issue in the L2 cache.

From the CPU L1 perspective here is what is happening :

• writeback 0x80019b00 (good value)
• wait the writeback is sent and then refill 0x80019b00
• refill got the old value of 0x80019b00 (bad)

=> L2 cache issue.

Looking at the wave, it seems the L2 cache you are using is older than a series of fix i did on it in SpinalHDL dev branch.
I think we need to update NaxRiscv with upstream SpinalHDL dev.
Can you give a try ?

Thanks ^^

[Dolu1990]

Forget what i said above.

Looking at the L2 cache, i can see that :

Where the 0x0000000 value is written by core 1 (source 6)
Then core 0 just read 0x00000000.

Are you sure the violation is on 0x80002550 and not on 0x80002560 ?

But yes, overall i think you are right, Nax L1 cache doesn't fully implements RVWMO CoRR in its LSU. There is no ordering load -> load, only between store -> load/store :/

[bantierr]

Hi, thank you so much for both comment!

Same address load-load ordering does not have to be enforced in RISC-V. Given that they return values from the same store, it is fine to re-order them, see the manual p.92:

a and b are loads, x is a byte read by both a and b, there is no store to x between a and b in
program order, and a and b return values for x written by different memory operations

And, I think you are right, it is the load at 0x80002560 that is the violation. In this case, what we break is the load axiom, because this load returns a values that is older that the latest store in global and local memory ordering. So a younger load (0x80002560) returns a value that is older than the value returned by an older load (0x80002550) in the same hart to the same address.

And, just for information, (p.643):

It is not the same as saying that the two loads return the same value, since two different stores may write the same value.

[bantierr]

Also, I already started updating to the latest dev commit of SpinalHDL. Form what I understand, I just have to update the submodule, re-apply the tracing patches, re-build and I am good to go?
And, is it fine to use the latest dev changes ?

Thank you again :)
Quentin

[Dolu1990]

Form what I understand, I just have to update the submodule, re-apply the tracing patches, re-build and I am good to go?

Probably yes, cd ext/SpinalHDL; git pull
Not 100% sure, i didn't tried since quite some time ^^

If anything fail let me know.

Regards
Charles

[Dolu1990]

@bantierr One question ^^
Which software / tool are you using to generate random test assembly programs ?
I'm interrested, in particular to test VexiiRiscv.

[bantierr]

I just wrote you an email to explain :)