The majority of the code change here is around TLS data stubs. The new TLS ABI requires that when relaxing a General Dynamic or Initial Exec access to a variable to a Local Exec access, the linker emits data stubs in a read-only section.
We do this with the below approach: - check_relocs notices that we need TLS data stubs by recognising that some relocation will need to be relaxed to a local-exec relocation. - check_relocs then records a hash table entry mapping the symbol that we are relocating against to a position in some data stub section. It also ensures that this data stub section has been created, and increments our data stub section size. - This section is placed in the resulting binary using the standard subsection and wildcard matching implemented by the generic linker. - In elfNN_aarch64_size_dynamic_sections we allocate the actual buffer for our data stub section. - When it comes to actually relaxing the TLS sequence, relocate_section directly populates the data stub using the address and size of the TLS object that has already been calculated, it then uses final_link_relocate to handle adjusting the text so that it points to this data stub.
Notes on implementation:
Mechanism by which we create and populate a TLS data section: There are currently three different ways by which the AArch64 backend creates and populates a special section. These are - The method by which the .got (and related sections) are populated. - The method by which the interworking stubs are populated. - The method by which erratum 843419 stub sections are populated.
We have gone with an approach that mostly follows that used to populate the .got. Here we give an outline of the approaches and provide the reasoning by which the approach used by the .got was chosen.
Handling the .got section: - Create a section on an existing BFD. - Mark that section as SEC_LINKER_CREATED. - Record the existing BFD as the `dynobj`. - bfd_elf_final_link still calls elf_link_input_bfd on the object. - elf_link_input_bfd avoids emitting the section (because of SEC_LINKER_CREATED). - bfd_elf_final_link then emits the special sections on `dynobj` after all the non-special sections on all objects have been relocated. - Allows updating the .got input section in relocate_section & final_link_relocate knowing that its contents will be output once all relocations on standard input sections have been processed.
Handling interworking stub sections. - Create a special stub file. - Create sections on that stub file for each input section we need a stub for. - Manually populate the sections in build_stubs (which is called through `ldemul_finish` *before* `ldwrite` and hence before any other files are relocated).
Handling erratum 843419 stub sections. - Create a special stub file. - Create sections on that stub file for each input section we need a stub for. - Ensure that the stub file is marked with class ELFCLASSNONE. - Ensure that the list of input sections for the relevant output section statement has the veneered input section *directly before* the stub section which has the veneer. - When relocating and outputting sections, having ELFCLASSNONE means that we output sections on the stub_file only when we see the corresponding input statement. Without that class marker bfd_elf_final_link calls elf_link_input_bfd which writes out the data for all input sections on the relevant BFD. - Since we have ensured the input statement for our stub section is directly after the input statement for the section we are emitting veneers for, we know that the veneered section will be relocated and output before we output our stub section. - Hence we can copy relocated data from the veneered section into our stub section and know that our stub section will be output after this modification has been made.
In deciding what to do with the read-only TLS data stubs we noticed the following problems with each approach: - The ABI requires that the read-only TLS data stubs are emitted into a read-only section. This will necessarily be a different output section to .text where the requirement for these stubs is found. The temporal order in which output sections are written to the output file is tied to the order in which the in-memory linker statements are kept, and that is tied to the linker script provided by the user. Hence we can not rely on ordering and ELFCLASSNONE to ensure that our data stub section is emitted after the relevant TLS sequences have been relaxed. (We need to know our data stub section is written to the output after we have populated it as otherwise the data would not propagate to the resulting binary). - I think it is easier and simpler to find the data needed for the TLS data stubs in relocate_section just as we relax the relevant TLS sequences. Hence I don't want to use the approach used for interworking stubs of populating the entire section beforehand. - Adding a section to `dynobj` would mean that we're adding a section to a user input BFD, which is not quite as clear as having a separate BFD for our special stub section. It also means we treat this particular section as a "dynamic" section. This is a little confusing nomenclature-wise.
Based on the above trade-offs we chose the .got approach (accepting the negative that this will be stored on a user BFD). N.b. using the .got approach and requiring the section get allocated in `size_dynamic_sections` is not problematic for static executables despite the nomenclature. This function always gets called.
One difference between how we handle the data stubs and how the .got is handled is that we do not count the number of data stubs required in size_dynamic_sections, but rather total it as we see relocations needing these stubs in check_relocs. We do this largely to avoid requiring another data member on all symbols to indicate information about whether this symbol needs a data stub and where that data stub is. The number of TLS symbols are expected to be much smaller than the number of symbols with an entry in the GOT and hence a separate hash table just containing entries for those symbols which need such information is likely to often be smaller.
N.b. it is interesting to mention that for all relocations which need a data stub we would make an input section on `dynobj` in `check_relocs` if that relaxation were not performed. This is since if we did not realise they could be relaxed these relocations would have needed a .got entry.
N.b. we must use make_section_anyway to create our TLS data stubs section in order to avoid any problems with our linker defined section having the same name as a section already defined by the user.
We do not use local stub symbols: The TLS ABI describes data stubs using specially named symbols. These are not part of the ABI. We could have associated the position of a data stub with a particular symbol by generating a symbol internally using some name mangling scheme that matches that in the TLS ABI examples and points to the data stub for a particular symbol. We take the current approach on the belief that it is "neater" to avoid relying on such a name-mangling scheme and the associated sprintf calls.
final_link_relocate handling the adjusted relocation for data stubs: final_link_relocate does not actually use the `h` or `sym` arguments to decide anything for the two relocations we need to handle once we have relaxed an IE or GD TLS access sequence to a LE one.
The relocations we need are BFD_RELOC_MORELLO_ADR_HI20_PCREL and BFD_RELOC_AARCH64_ADD_LO12. For both (and in fact for most relocations) we only use `h` and `sym` for catching and reporting errors.
This means we don't actually have to update the `h` and/or `sym` variables before calling elfNN_aarch64_final_link_relocate.
Allocate TLS data stubs on dynobj This uses the same approach that the linker uses to ensure that the .got sections are emitted after all relocations have been processed.
elf_link_input_bfd avoids sections with SEC_LINKER_CREATED, and bfd_elf_final_link emits all SEC_LINKER_CREATED sections on the dynobj *after* standard sections have been relocated.
This means that we can populate the contents of the TLS data stub section while performing relocations on all our other sections (i.e. in the same place as we perform the relaxations on the TLS sequences that we recognise need these data stubs).
Assert that copy_indirect_symbol is not a problem copy_indirect_symbol takes information from one symbol and puts it onto another. The point is to ensure that any symbol which simply refers to another has all its cached information on that symbol to which it refers rather than itself. If we could ever call this function on a symbol which we have found needs an associated data stub created, then we could have to handle adjusting the hash table associating a symbol with a data stub. We do not believe this is needed, and add an assert instead.
The proof that this is not a problem is a little tricky. However it *shouldn't* be a problem given what it's handling. This is handling moving cached information from an indirected symbol to the symbol it represents. That is needed when the information was originally put on the indirected symbol, and that happens when the indirection was originally the other way around. The two ways that this reversal of indirection can happen is through resolving dynamic weak symbols and versioned symbols. Both of these are not something we can see with SYMBOL_REFERENCES_LOCAL TLS symbols (see below).
We only need to worry about copy_indirect_symbol transferring information *from* a symbol which we have generated a TLS relaxation against to LE.
In order to satisfy the criteria that we have generated a TLS relaxation hash entry against a symbol, we must have already have run check_relocs. This means that of the ways in which copy_indirect_symbol can be called we have eliminated all but _bfd_elf_fix_symbol_flags and bfd_elf_record_link_assignment.
bfd_elf_record_link_assignment handles symbol assignments in a linker script. Such assignments can not be made on TLS symbols (we end up generating a non-TLS symbol).
_bfd_elf_fix_symbol_flags only calls copy_indirect_symbol on symbols which have is_weakalias set. These are symbols "from a dynamic object", and we only ever call the hook when the real definition is in a non-shared object. Hence we would not have performed this relaxation on the symbol (because it is not SYMBOL_REFERENCES_LOCAL).
Hence I don't believe this is something that we can trigger and we add an assertion here rather than add code to handle the case.
############### Attachment also inlined for ease of reply ###############
diff --git a/bfd/elfnn-aarch64.c b/bfd/elfnn-aarch64.c index 7e951a5da73c3f427bbcf29093dda521e86d5e75..ac357c4fa82c910892fe1c46a91f173883c19df8 100644 --- a/bfd/elfnn-aarch64.c +++ b/bfd/elfnn-aarch64.c @@ -341,7 +341,8 @@ relocation for it when we can get an untagged zero capability by just loading some zeros. */ #define c64_needs_relocation(info, h) \ - (!((h)->root.type == bfd_link_hash_undefweak \ + (!((h) \ + && (h)->root.type == bfd_link_hash_undefweak \ && (UNDEFWEAK_NO_DYNAMIC_RELOC ((info), (h)) \ || !elf_hash_table ((info))->dynamic_sections_created)))
@@ -3032,6 +3033,19 @@ struct elf_aarch64_stub_hash_entry bfd_vma adrp_offset; };
+struct elf_c64_tls_data_stub_hash_entry +{ + /* Symbol table entry, if any, for which this stub is made. */ + struct elf_aarch64_link_hash_entry *h; + /* Local symbol table index and BFD associated with it, if required, for + which this stub is made. These are only used when `h` is NULL. */ + unsigned long r_symndx; + bfd *input_bfd; + /* Offset within htab->sc64_tls_stubs of the beginning of this stub. */ + bfd_vma tls_stub_offset; + bfd_boolean populated; +}; + /* Used to build a map of a section. This is required for mixed-endian code/data. */
@@ -3327,8 +3341,11 @@ struct elf_aarch64_link_hash_table
/* Used for capability relocations. */ asection *srelcaps; + asection *sc64_tls_stubs; int c64_rel; bfd_boolean c64_output; + htab_t c64_tls_data_stub_hash_table; + void * tls_data_stub_memory; };
/* Create an entry in an AArch64 ELF linker hash table. */ @@ -3402,6 +3419,119 @@ stub_hash_newfunc (struct bfd_hash_entry *entry, return entry; }
+static hashval_t +c64_tls_data_stub_hash (const void *ptr) +{ + struct elf_c64_tls_data_stub_hash_entry *entry + = (struct elf_c64_tls_data_stub_hash_entry *) ptr; + if (entry->h) + return htab_hash_pointer (entry->h); + struct elf_aarch64_local_symbol *l; + l = elf_aarch64_locals (entry->input_bfd); + return htab_hash_pointer (l + entry->r_symndx); +} + +static int +c64_tls_data_stub_eq (const void *ptr1, const void *ptr2) +{ + struct elf_c64_tls_data_stub_hash_entry *entry1 + = (struct elf_c64_tls_data_stub_hash_entry *) ptr1; + struct elf_c64_tls_data_stub_hash_entry *entry2 + = (struct elf_c64_tls_data_stub_hash_entry *) ptr2; + + if (entry1->h && !entry2->h) + return 0; + if (!entry1->h && entry2->h) + return 0; + if (entry1->h && entry2->h) + return entry1->h == entry2->h; + + if (entry1->input_bfd != entry2->input_bfd) + return 0; + return entry1->r_symndx == entry2->r_symndx; +} + +static bfd_boolean +c64_record_tls_stub (struct elf_aarch64_link_hash_table *htab, + bfd *input_bfd, + struct elf_link_hash_entry *h, + unsigned long r_symndx) +{ + if (htab->root.dynobj == NULL) + htab->root.dynobj = input_bfd; + + if (!htab->sc64_tls_stubs) + { + asection *stub_sec; + flagword flags; + + flags = (SEC_ALLOC | SEC_LOAD | SEC_READONLY + | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_KEEP + | SEC_LINKER_CREATED); + /* Using the .rodata section is ABI -- N.b. I do have an outstanding + clarification on this on the ABI PR. + https://github.com/ARM-software/abi-aa/pull/80 */ + stub_sec = bfd_make_section_anyway_with_flags + (htab->root.dynobj, ".rodata", flags); + if (stub_sec == NULL) + return FALSE; + + /* Section contains stubs of 64 bit values, hence requires 8 byte + alignment. */ + bfd_set_section_alignment (stub_sec, 3); + htab->sc64_tls_stubs = stub_sec; + BFD_ASSERT (htab->sc64_tls_stubs->size == 0); + } + + if (h) + BFD_ASSERT (h->type == STT_TLS); + + void **slot; + struct elf_c64_tls_data_stub_hash_entry e, *new_entry; + e.h = (struct elf_aarch64_link_hash_entry *)h; + e.r_symndx = r_symndx; + e.input_bfd = input_bfd; + + slot = htab_find_slot (htab->c64_tls_data_stub_hash_table, &e, INSERT); + if (!slot) + return FALSE; + + if (*slot) + return TRUE; + new_entry = (struct elf_c64_tls_data_stub_hash_entry *) + objalloc_alloc ((struct objalloc *) htab->tls_data_stub_memory, + sizeof (struct elf_c64_tls_data_stub_hash_entry)); + if (new_entry) + { + new_entry->h = (struct elf_aarch64_link_hash_entry *)h; + new_entry->r_symndx = r_symndx; + new_entry->input_bfd = input_bfd; + new_entry->tls_stub_offset = htab->sc64_tls_stubs->size; + new_entry->populated = FALSE; + /* Size of a Morello data stub is 2 * 8 byte integers. */ + htab->sc64_tls_stubs->size += 16; + *slot = new_entry; + return TRUE; + } + return FALSE; +} + +static struct elf_c64_tls_data_stub_hash_entry * +c64_tls_stub_find (struct elf_link_hash_entry *h, + bfd *input_bfd, + unsigned long r_symndx, + struct elf_aarch64_link_hash_table *htab) +{ + void *ret; + struct elf_c64_tls_data_stub_hash_entry e; + e.h = (struct elf_aarch64_link_hash_entry *)h; + e.r_symndx = r_symndx; + e.input_bfd = input_bfd; + + ret = htab_find (htab->c64_tls_data_stub_hash_table, &e); + return (struct elf_c64_tls_data_stub_hash_entry *)ret; +} + /* Compute a hash of a local hash entry. We use elf_link_hash_entry for local symbol so that we can handle local STT_GNU_IFUNC symbols as global symbol. We reuse indx and dynstr_index for local symbol @@ -3491,6 +3621,13 @@ elfNN_aarch64_copy_indirect_symbol (struct bfd_link_info *info, } }
+ /* This function is called to take the indirect symbol information from eind + and put it onto edir. We never create a TLS data stub for a symbol which + needs such a transformation, hence there is no need to worry about + removing the hash entry for the indirected symbol and ensuring there is + now one for the final one. */ + BFD_ASSERT (!c64_tls_stub_find (ind, NULL, 0, + elf_aarch64_hash_table (info))); _bfd_elf_link_hash_copy_indirect (info, dir, ind); }
@@ -3532,6 +3669,11 @@ elfNN_aarch64_link_hash_table_free (bfd *obfd) if (ret->loc_hash_memory) objalloc_free ((struct objalloc *) ret->loc_hash_memory);
+ if (ret->c64_tls_data_stub_hash_table) + htab_delete (ret->c64_tls_data_stub_hash_table); + if (ret->tls_data_stub_memory) + objalloc_free ((struct objalloc *) ret->tls_data_stub_memory); + bfd_hash_table_free (&ret->stub_hash_table); _bfd_elf_link_hash_table_free (obfd); } @@ -3571,6 +3713,16 @@ elfNN_aarch64_link_hash_table_create (bfd *abfd) return NULL; }
+ ret->c64_tls_data_stub_hash_table + = htab_try_create (256, c64_tls_data_stub_hash, c64_tls_data_stub_eq, + NULL); + ret->tls_data_stub_memory = objalloc_create (); + if (!ret->c64_tls_data_stub_hash_table || !ret->tls_data_stub_memory) + { + elfNN_aarch64_link_hash_table_free (abfd); + return NULL; + } + ret->loc_hash_table = htab_try_create (1024, elfNN_aarch64_local_htab_hash, elfNN_aarch64_local_htab_eq, @@ -5992,18 +6144,13 @@ static bfd_reloc_code_real_type aarch64_tls_transition_without_check (bfd_reloc_code_real_type r_type, struct bfd_link_info *info, struct elf_link_hash_entry *h, - bfd_boolean morello_reloc) + bfd_boolean *requires_c64_tls_stub) { bfd_boolean local_exec = (bfd_link_executable (info) && TLS_SYMBOL_REFERENCES_LOCAL (info, h));
switch (r_type) { - case BFD_RELOC_MORELLO_TLSDESC_ADR_PAGE20: - return (local_exec - ? BFD_RELOC_AARCH64_TLSLE_MOVW_TPREL_G1 - : r_type); - case BFD_RELOC_AARCH64_TLSDESC_ADR_PAGE21: case BFD_RELOC_AARCH64_TLSGD_ADR_PAGE21: return (local_exec @@ -6035,16 +6182,24 @@ aarch64_tls_transition_without_check (bfd_reloc_code_real_type r_type, ? BFD_RELOC_AARCH64_TLSLE_MOVW_TPREL_G2 : BFD_RELOC_AARCH64_TLSIE_MOVW_GOTTPREL_G1);
- case BFD_RELOC_MORELLO_TLSDESC_LD128_LO12: - return (local_exec - ? BFD_RELOC_AARCH64_TLSLE_MOVW_TPREL_G0_NC : r_type); - case BFD_RELOC_AARCH64_TLSDESC_LDNN_LO12_NC: case BFD_RELOC_AARCH64_TLSGD_ADD_LO12_NC: return (local_exec ? BFD_RELOC_AARCH64_TLSLE_MOVW_TPREL_G0_NC : BFD_RELOC_AARCH64_TLSIE_LDNN_GOTTPREL_LO12_NC);
+ case BFD_RELOC_MORELLO_TLSIE_ADR_GOTTPREL_PAGE20: + if (!local_exec) + return r_type; + *requires_c64_tls_stub = TRUE; + return BFD_RELOC_MORELLO_ADR_HI20_PCREL; + + case BFD_RELOC_MORELLO_TLSIE_ADD_LO12: + if (!local_exec) + return r_type; + *requires_c64_tls_stub = TRUE; + return BFD_RELOC_AARCH64_ADD_LO12; + case BFD_RELOC_AARCH64_TLSIE_ADR_GOTTPREL_PAGE21: return local_exec ? BFD_RELOC_AARCH64_TLSLE_MOVW_TPREL_G1 : r_type;
@@ -6059,19 +6214,32 @@ aarch64_tls_transition_without_check (bfd_reloc_code_real_type r_type, ? BFD_RELOC_AARCH64_TLSLE_ADD_TPREL_HI12 : BFD_RELOC_AARCH64_TLSIE_LD_GOTTPREL_PREL19);
+ case BFD_RELOC_MORELLO_TLSDESC_ADR_PAGE20: + if (local_exec) + { + *requires_c64_tls_stub = TRUE; + return BFD_RELOC_MORELLO_ADR_HI20_PCREL; + } + return BFD_RELOC_MORELLO_TLSIE_ADR_GOTTPREL_PAGE20; + + case BFD_RELOC_MORELLO_TLSDESC_LD128_LO12: + if (local_exec) + { + *requires_c64_tls_stub = TRUE; + return BFD_RELOC_AARCH64_ADD_LO12; + } + return BFD_RELOC_MORELLO_TLSIE_ADD_LO12; + case BFD_RELOC_MORELLO_TLSDESC_CALL: - return (local_exec - ? BFD_RELOC_AARCH64_NONE : r_type); + /* Instructions with this relocation will be fully resolved during the + transition into an add and scbnds pair. */ + return BFD_RELOC_AARCH64_NONE;
case BFD_RELOC_AARCH64_TLSDESC_ADD_LO12: - if (morello_reloc && !local_exec) - return r_type; - /* Fall through. */ case BFD_RELOC_AARCH64_TLSDESC_ADD: case BFD_RELOC_AARCH64_TLSDESC_CALL: /* Instructions with these relocations will be fully resolved during the - transition into either a NOP in the A64 case or movk and add in - C64. */ + transition into either a NOP in the A64 case or an ldp in C64. */ return BFD_RELOC_AARCH64_NONE;
case BFD_RELOC_AARCH64_TLSLD_ADD_LO12_NC: @@ -6200,8 +6368,12 @@ aarch64_tls_transition (bfd *input_bfd, struct bfd_link_info *info, const Elf_Internal_Rela *rel, struct elf_link_hash_entry *h, - unsigned long r_symndx) + unsigned long r_symndx, + bfd_boolean *requires_c64_tls_stub) { + /* Initialisation done here. The set to TRUE is done in + aarch64_tls_transition_without_check if necessary. */ + *requires_c64_tls_stub = FALSE; bfd_reloc_code_real_type bfd_r_type = elfNN_aarch64_bfd_reloc_from_type (input_bfd, ELFNN_R_TYPE (rel->r_info)); @@ -6209,12 +6381,8 @@ aarch64_tls_transition (bfd *input_bfd, if (! aarch64_can_relax_tls (input_bfd, info, rel, h, r_symndx)) return bfd_r_type;
- bfd_boolean morello_reloc = (bfd_r_type == BFD_RELOC_AARCH64_TLSDESC_ADD_LO12 - && (ELFNN_R_TYPE (rel[1].r_info) - == MORELLO_R (TLSDESC_CALL))); - return aarch64_tls_transition_without_check (bfd_r_type, info, h, - morello_reloc); + requires_c64_tls_stub); }
/* Return the base VMA address which should be subtracted from real addresses @@ -7841,7 +8009,7 @@ elfNN_aarch64_final_link_relocate (reloc_howto_type *howto,
outrel.r_addend = signed_addend;
- if (h && !c64_needs_relocation (info, h)) + if (!c64_needs_relocation (info, h)) { /* If we know this symbol does not need a C64 dynamic relocation then it must be because this is an undefined weak symbol which @@ -7952,6 +8120,12 @@ elfNN_aarch64_final_link_relocate (reloc_howto_type *howto, # define movz_hw_R0 (0x52c00000) #endif
+/* C64 only instructions. */ +#define add_C0_C0 (0x02000000) +#define ldp_X0_X1_C0 (0xa9400400) +#define add_C0_C2_X0 (0xc2a06040) +#define scbnds_C0_C0_X1 (0xc2c10000) + /* Structure to hold payload for _bfd_aarch64_erratum_843419_clear_stub, it is used to identify the stub information to reset. */
@@ -8027,53 +8201,18 @@ static bfd_reloc_status_type elfNN_aarch64_tls_relax (bfd *input_bfd, struct bfd_link_info *info, asection *input_section, bfd_byte *contents, Elf_Internal_Rela *rel, - struct elf_link_hash_entry *h, unsigned long r_symndx) + struct elf_link_hash_entry *h) { bfd_boolean local_exec = (bfd_link_executable (info) && TLS_SYMBOL_REFERENCES_LOCAL (info, h)); unsigned int r_type = ELFNN_R_TYPE (rel->r_info); unsigned long insn; - bfd_vma sym_size = 0; struct elf_aarch64_link_hash_table *globals = elf_aarch64_hash_table (info);
BFD_ASSERT (globals && input_bfd && contents && rel);
- if (local_exec) - { - if (h != NULL) - sym_size = h->size; - else - { - Elf_Internal_Sym *sym; - - sym = bfd_sym_from_r_symndx (&globals->root.sym_cache, input_bfd, - r_symndx); - BFD_ASSERT (sym != NULL); - sym_size = sym->st_size; - } - } - switch (elfNN_aarch64_bfd_reloc_from_type (input_bfd, r_type)) { - case BFD_RELOC_MORELLO_TLSDESC_ADR_PAGE20: - if (local_exec) - { - /* GD->LE relaxation: - nop => movz x1, objsize_hi16 - adrp x0, :tlsdesc:var => movz x0, :tprel_g1:var */ - bfd_putl32 (BUILD_MOVZ(1, sym_size), contents + rel->r_offset - 4); - bfd_putl32 (movz_R0, contents + rel->r_offset); - - /* We have relaxed the adrp into a mov, we may have to clear any - pending erratum fixes. */ - clear_erratum_843419_entry (globals, rel->r_offset, input_section); - return bfd_reloc_continue; - } - else - { - /* GD->IE relaxation: Not implemented. */ - return bfd_reloc_continue; - } case BFD_RELOC_AARCH64_TLSDESC_ADR_PAGE21: case BFD_RELOC_AARCH64_TLSGD_ADR_PAGE21: if (local_exec) @@ -8239,22 +8378,21 @@ elfNN_aarch64_tls_relax (bfd *input_bfd, struct bfd_link_info *info, return bfd_reloc_continue; #endif
+ case BFD_RELOC_MORELLO_TLSIE_ADR_GOTTPREL_PAGE20: + /* IE->LE relaxation: + adrp c0, :gottprel:var => adrp c0, __var_data + Instruction does not change (just the relocation on it). */ + return bfd_reloc_continue; + + case BFD_RELOC_MORELLO_TLSIE_ADD_LO12: + /* IE->LE relaxation: + add c0, c0, :gottprel_lo12:var => add c0, c0, :lo12:__var_data + Instruction does not change (just the relocation on it). */ + return bfd_reloc_continue; + case BFD_RELOC_AARCH64_TLSIE_LD_GOTTPREL_PREL19: return bfd_reloc_continue;
- case BFD_RELOC_MORELLO_TLSDESC_LD128_LO12: - if (local_exec) - { - /* GD->LE relaxation: - ldr xd, [x0, #:tlsdesc_lo12:var] => movk x0, :tprel_g0_nc:var */ - bfd_putl32 (movk_R0, contents + rel->r_offset); - return bfd_reloc_continue; - } - else - { - /* GD->IE relaxation: not implemented. */ - return bfd_reloc_continue; - } case BFD_RELOC_AARCH64_TLSDESC_LDNN_LO12_NC: if (local_exec) { @@ -8319,25 +8457,39 @@ elfNN_aarch64_tls_relax (bfd *input_bfd, struct bfd_link_info *info, return bfd_reloc_continue; }
- case BFD_RELOC_MORELLO_TLSDESC_CALL: + case BFD_RELOC_MORELLO_TLSDESC_ADR_PAGE20: + /* GD->IE relaxation: + adrp c0, :tlsdesc:var => adrp c0, :gottprel:var + GD->LE relaxation: + adrp c0, :tlsdesc:var => adrp c0, __var_data + (No change in instructions, just need a different relocation). */ + return bfd_reloc_continue; + + case BFD_RELOC_MORELLO_TLSDESC_LD128_LO12: /* GD->LE relaxation: - blr cd => add c0, c2, x0 */ - if (local_exec) - { - bfd_putl32 (0xc2a06040, contents + rel->r_offset); - return bfd_reloc_ok; - } - else - goto set_nop; + ldr c1, [c0, #:tlsdesc_lo12:var] => add c0, c0, :lo12:__var_data + GD->IE relaxation: + ldr c1, [c0, #:tlsdesc_lo12:var] => add c0, c0, :gottprel_lo12:__var_data + (same relaxation in terms of instruction change, only different in + terms of relocation that is used). */ + bfd_putl32 (add_C0_C0, contents + rel->r_offset); + return bfd_reloc_continue; + + case BFD_RELOC_MORELLO_TLSDESC_CALL: + /* GD->LE relaxation AND GD->IE relaxation: + nop => add c0, c2, x0 + blr cd => scbnds c0, c0, x1 */ + bfd_putl32 (add_C0_C2_X0, contents + rel->r_offset - 4); + bfd_putl32 (scbnds_C0_C0_X1, contents + rel->r_offset); + return bfd_reloc_ok;
case BFD_RELOC_AARCH64_TLSDESC_ADD_LO12: - /* GD->LE relaxation: - ldr cd, [c0, #:tlsdesc_lo12:var] => movk x1, objsize_lo16 */ - if (local_exec - && ELFNN_R_TYPE (rel[1].r_info) == MORELLO_R (TLSDESC_CALL)) + /* GD->LE relaxation AND GD->IE relaxation: + add c0, c0, :tlsdesc_lo12:var => ldp x0, x1, [c0] */ + if (ELFNN_R_TYPE (rel[1].r_info) == MORELLO_R (TLSDESC_CALL)) { - bfd_putl32 (BUILD_MOVK(1, sym_size), contents + rel->r_offset); - return bfd_reloc_continue; + bfd_putl32 (ldp_X0_X1_C0, contents + rel->r_offset); + return bfd_reloc_ok; }
/* Fall through. */ @@ -8347,7 +8499,6 @@ elfNN_aarch64_tls_relax (bfd *input_bfd, struct bfd_link_info *info, add x0, x0, #:tlsdesc_lo12:var => nop blr xd => nop */ -set_nop: bfd_putl32 (INSN_NOP, contents + rel->r_offset); return bfd_reloc_ok;
@@ -8489,6 +8640,35 @@ set_nop:
/* Relocate an AArch64 ELF section. */
+static bfd_vma +c64_populate_tls_data_stub (struct elf_aarch64_link_hash_table *globals, + bfd *input_bfd, + struct elf_link_hash_entry *h, + unsigned long r_symndx, + bfd_vma offset, bfd_vma size, + bfd *output_bfd) +{ + struct elf_c64_tls_data_stub_hash_entry *found + = c64_tls_stub_find (h, input_bfd, r_symndx, globals); + BFD_ASSERT (found); + + if (!found->populated) + { + bfd_put_NN (output_bfd, offset, + globals->sc64_tls_stubs->contents + + found->tls_stub_offset); + bfd_put_NN (output_bfd, size, + globals->sc64_tls_stubs->contents + + found->tls_stub_offset + 8); + found->populated = TRUE; + } + + return globals->sc64_tls_stubs->output_section->vma + + globals->sc64_tls_stubs->output_offset + + found->tls_stub_offset; +} + + static bfd_boolean elfNN_aarch64_relocate_section (bfd *output_bfd, struct bfd_link_info *info, @@ -8643,16 +8823,26 @@ elfNN_aarch64_relocate_section (bfd *output_bfd, We call elfNN_aarch64_final_link_relocate unless we're completely done, i.e., the relaxation produced the final output we want. */
+ bfd_boolean requires_c64_tls_stub; relaxed_bfd_r_type = aarch64_tls_transition (input_bfd, info, rel, - h, r_symndx); + h, r_symndx, + &requires_c64_tls_stub); if (relaxed_bfd_r_type != bfd_r_type) { bfd_r_type = relaxed_bfd_r_type; howto = elfNN_aarch64_howto_from_bfd_reloc (bfd_r_type); BFD_ASSERT (howto != NULL); r_type = howto->type; + if (requires_c64_tls_stub) + { + relocation + = c64_populate_tls_data_stub (globals, input_bfd, h, r_symndx, + relocation - tpoff_base (info), + h ? h->size : sym->st_size, + output_bfd); + } r = elfNN_aarch64_tls_relax (input_bfd, info, input_section, - contents, rel, h, r_symndx); + contents, rel, h); unresolved_reloc = 0; } else @@ -8717,7 +8907,7 @@ elfNN_aarch64_relocate_section (bfd *output_bfd,
loc = globals->root.srelgot->contents; loc += globals->root.srelgot->reloc_count++ - * RELOC_SIZE (htab); + * RELOC_SIZE (globals); bfd_elfNN_swap_reloca_out (output_bfd, &rela, loc);
bfd_reloc_code_real_type real_type = @@ -8779,6 +8969,9 @@ elfNN_aarch64_relocate_section (bfd *output_bfd,
case BFD_RELOC_MORELLO_TLSIE_ADR_GOTTPREL_PAGE20: case BFD_RELOC_MORELLO_TLSIE_ADD_LO12: + c64_rtype = TRUE; + /* Fall through. */ + case BFD_RELOC_AARCH64_TLSIE_ADR_GOTTPREL_PAGE21: case BFD_RELOC_AARCH64_TLSIE_LDNN_GOTTPREL_LO12_NC: case BFD_RELOC_AARCH64_TLSIE_LD_GOTTPREL_PREL19: @@ -8800,6 +8993,7 @@ elfNN_aarch64_relocate_section (bfd *output_bfd, (h == NULL || ELF_ST_VISIBILITY (h->other) == STV_DEFAULT || h->root.type != bfd_link_hash_undefweak); + BFD_ASSERT (!c64_rtype || c64_needs_relocation (info, h)); need_relocs = need_relocs || c64_rtype;
BFD_ASSERT (globals->root.srelgot != NULL); @@ -8822,7 +9016,7 @@ elfNN_aarch64_relocate_section (bfd *output_bfd,
loc = globals->root.srelgot->contents; loc += globals->root.srelgot->reloc_count++ - * RELOC_SIZE (htab); + * RELOC_SIZE (globals);
bfd_elfNN_swap_reloca_out (output_bfd, &rela, loc);
@@ -8864,6 +9058,7 @@ elfNN_aarch64_relocate_section (bfd *output_bfd, need_relocs = (h == NULL || ELF_ST_VISIBILITY (h->other) == STV_DEFAULT || h->root.type != bfd_link_hash_undefweak); + BFD_ASSERT (!c64_rtype || c64_needs_relocation (info, h)); need_relocs = need_relocs || c64_rtype;
BFD_ASSERT (globals->root.srelgot != NULL); @@ -9562,7 +9757,16 @@ elfNN_aarch64_check_relocs (bfd *abfd, struct bfd_link_info *info, }
/* Could be done earlier, if h were already available. */ - bfd_r_type = aarch64_tls_transition (abfd, info, rel, h, r_symndx); + bfd_boolean requires_c64_tls_stub; + bfd_r_type = aarch64_tls_transition (abfd, info, rel, h, r_symndx, + &requires_c64_tls_stub); + if (requires_c64_tls_stub + && !c64_record_tls_stub (htab, abfd, h, r_symndx)) + { + _bfd_error_handler (_("%pB: failed to record TLS stub"), abfd); + bfd_set_error (bfd_error_no_memory); + return FALSE; + }
if (h != NULL) { @@ -9950,7 +10154,7 @@ elfNN_aarch64_check_relocs (bfd *abfd, struct bfd_link_info *info, break;
case BFD_RELOC_MORELLO_CAPINIT: - if (h && !c64_needs_relocation (info, h)) + if (!c64_needs_relocation (info, h)) /* If this symbol does not need a relocation, then there's no reason to increase the srelcaps size for a relocation. */ break; @@ -10565,14 +10769,17 @@ elfNN_aarch64_allocate_dynrelocs (struct elf_link_hash_entry *h, void *inf) htab->root.sgot->size += GOT_ENTRY_SIZE (htab); }
+ /* We avoid TLS relocations on undefweak symbols since it is not + well defined. Hence we should not be seeing got entries + on any symbol which would not need a relocation in a C64 binary. + */ + BFD_ASSERT (c64_needs_relocation (info, h)); indx = h && h->dynindx != -1 ? h->dynindx : 0; if ((ELF_ST_VISIBILITY (h->other) == STV_DEFAULT || h->root.type != bfd_link_hash_undefweak) && (!bfd_link_executable (info) || indx != 0 || WILL_CALL_FINISH_DYNAMIC_SYMBOL (dyn, 0, h) - /* On Morello support only TLSDESC_GD to TLSLE relaxation; - for everything else we must emit a dynamic relocation. */ || htab->c64_rel)) { if (got_type & GOT_TLSDESC_GD) @@ -10980,7 +11187,8 @@ elfNN_aarch64_size_dynamic_sections (bfd *output_bfd, || s == htab->root.iplt || s == htab->root.igotplt || s == htab->root.sdynbss - || s == htab->root.sdynrelro) + || s == htab->root.sdynrelro + || s == htab->sc64_tls_stubs) { /* Strip this section if we don't need it; see the comment below. */