You might be thinking that the Br side is -1 and the H side is +1. That is certainly not the case. Bromine has an oxidation state of -1, while the oxidation state of hydrogen is +1. Oxidation states are assigned numbers and are NOT the actual charges on atoms in compounds (neither are formal charges).
The actual charges are derived from the electronegativity difference. The charge at the H end of the molecule is around +0.19, and the charge at the Br end of the molecule is around -0.19. http://www.colby.edu/chemistry/webmo/HBr.html