The simple answer to your question is ‘choice’.
Your grandfather chose to emigrate to America for economic (or political) reasons, whereas Africans were hunted, captured, shackled and transported to the Americas and sold into slavery against their will.
Even when slavery was abolished their decedents were ‘othered’ and lived under an apartheid regime in large swathes of the US.
Whilst other immigrant populations undoubtedly faced discrimination, no white immigrants suffered apartheid in the same way.
Jim Crow laws continued to ‘other’ black Americans right up until 1965 - that’s within my lifetime ( and many others on OF. )
So black Americans didn’t segregate themselves but were forced into a defined demographic against their will and, whilst not entirely ideal for the promotion of a cohesive society, it is entirely understandable why they might now choose to embrace that label for themselves.