Defending visit, chairman of national Holocaust memorial and museum in Jerusalem says the site is open to all and ‘does not engage’ with German party’s members