Abstract: Phase generated carrier (PGC) demodulation method based on ellipse fitting is an effective method to eliminate the influence of nonlinear factors on the PGC demodulation results of optical fiber hydrophone. The optimal estimation of elliptic curve parameters is the key to realizing this method. Extended Kalman particle filtering (EPF) algorithm is a commonly used optimal estimation algorithm to solve this kind of nonlinear problem. When the traditional EPF algorithm is used for parameter or state estimation of process equations with constant parameters, the variance of process noise is usually set to a constant, which makes the algorithm difficult to take into account the convergence speed and estimation accuracy, and limits the overall performance of the algorithm. In order to solve this problem, the existing EPF algorithm is improved in this paper, and an adaptive extended Kalman particle filtering (AEPF) algorithm is proposed. The results of the simulation and experiment show that the AEPF algorithm proposed in this paper can effectively estimate the parameters of elliptic curve, and can effectively demodulate the acoustic signal to be measured according to the PGC demodulation method of optical fiber hydrophone based on elliptic fitting mentioned in this paper. And the AEPF algorithm has higher convergence rate and higher accuracy than the EKF and EPF algorithms. It should be noted that the AEPF algorithm proposed in this paper is also applicable to other parameter or state estimation problems with constant parameter process equations, and has certain universality.