Existing adaptive observers may suffer parameter estimate drift due to disturbances even if state estimation errors remain small. To avoid such drift in the presence of bounded disturbances, several robust adaptive observers have been introduced providing bounds in state and parameter estimates. However, it is not easy for these observers to manipulate the size of the bounds with the selection of the observer gain. To reduce estimation errors, this paper introduces the H-infinity norm minimization problem in the adaptive observer structure, which minimizes the H-infinity norm between disturbances and estimation errors. The stability condition of the adaptive observer is reformulated as a linear matrix inequality, and the observer gain is optimally chosen by solving the resulting convex optimization problem. The estimation performance is demonstrated through a numerical example.