Do you think the "RC Layer Direct Sum Hypothesis" is valid? Reasons supporting the validity of the "RC Layer Direct Sum Hypothesis" include: ① Empirical Evidence: Functional Separation of ...

GPT models retain "meaning knowledge" as long-term memory in FFN (especially NL layers) and as short-term memory in SA (especially memory {Vi} layers). In the transformer layer's [ SA-RC → (LN-FFN)-RC ...