Behavior that confounds veterans of the rare Jewish book market has become routine at auctions organized by Genazym as it taps into emerging luxury market among Orthodox Jews