Using some creativity, I used a chisel to split the remaining piece in the block. That turned out okay but it also pushed it into the hole. Then using a pair of needle nose pliers, I managed to distort the remainder and pull it free of the block. Just for grins, I ran a magnet around in the hole. The new piece is shown here for comparison.