Hi Dietmar,

On 2021/9/9 16:34, Dietmar Eggemann wrote:

Hi,

We are carrying out some NUMA related test activities (some of which we discussed on this forum) on the Taishan 2280 v2 server.

For those tests we need memory attached to all 4 NUMA nodes equally. That's why we added 2 extra 32GB memory modules to the existing 2.

The manual says that they should be placed into slot:

000 (CPU A) 100 (CPU A) 020 (CPU B) and 120 (CPU B).

BMC console as well as startup logs show that the memory modules get properly detected in these slots.

But after Linux booted we don't have the expected 32 GB per NUMA node. There is node 1 & 3 with 64GB and node 0 & 2 with no memory instead.

We didn't find any BIOS options which could explain this memory distribution.

Do you have any hints why this is happening? Any help with this issue is highly appreciated.

Maybe it's related to the VA_BITS you set in the kernel config, so if you using 39 bits VA, it will not cover all the PA ranges reported by the firmware, so please set VA_BITS to 42 or 48, such as CONFIG_ARM64_VA_BITS_48=y.

Please let me know if it works.

Thanks Hanjun

Thanks Barry, I think you are right.

I misunderstood Dietmar's feedback and thought there are 64 GB memory before placing the 32G memory.

Thanks Hanjun

On 2021/9/10 14:53, Song Bao Hua (Barry Song) via Linaro-open-discussions wrote:

Hi Dietmar, Nice to hear from you.

Each cpu socket has two DIEs(two NUMA). It seems you are placing all memories in one DIE of one socket.

You might want to place one memory on odd slots like 010, 030. 0 EVEN 0 -> one numa 0 ODD 0 -> the other numa.

BTW, you might want to put one memory in each memory slot or place half-full memories(one memory in each memory channel). Otherwise, the bandwidth would be limited and the performance might behave differently with a system whose memories are fully or half-fully installed.

Thanks Barry

-----Original Message----- From: Linaro-open-discussions [mailto:linaro-open-discussions-bounces@op-lists.linaro.org] On Behalf Of Dietmar Eggemann via Linaro-open-discussions Sent: Thursday, September 9, 2021 8:34 PM To: linaro-open-discussions@op-lists.linaro.org Subject: [Linaro-open-discussions] Memory distribution between NUMA nodes on Taishan 2280 v2

Hi,

We are carrying out some NUMA related test activities (some of which we discussed on this forum) on the Taishan 2280 v2 server.

For those tests we need memory attached to all 4 NUMA nodes equally. That's why we added 2 extra 32GB memory modules to the existing 2.

The manual says that they should be placed into slot:

000 (CPU A) 100 (CPU A) 020 (CPU B) and 120 (CPU B).

BMC console as well as startup logs show that the memory modules get properly detected in these slots.

But after Linux booted we don't have the expected 32 GB per NUMA node. There is node 1 & 3 with 64GB and node 0 & 2 with no memory instead.

We didn't find any BIOS options which could explain this memory distribution.

Do you have any hints why this is happening? Any help with this issue is highly appreciated.

Thanks!

-- Dietmar