In recent years, support vector machine (SVM) has been widely used for supervised classification and yielded lots of exciting results within hyperspectral images (HSI) community. It seeks the optimal hyperplane, which maximizes the margin between two investigated classes. In the existing works, spatial feature usually gets involved as input and shows excellent classification performance. Unlike those available where feature generation is independent with a classifier, in our work, a hybrid framework that couples SVM classifier training with Gabor feature learning for joint optimization is proposed. It is called hybrid SVM (HSVM). Along with HSVM, two data-driven schemes are further designed. For one thing, Kronecker decomposition is applied to convert the feature learning part into a bilinear form. It aims to discover the interfilter structure characteristic of Gabor feature. For another thing, a local regularization term is appended to capture discriminant information as well. The iterative strategy is employed to optimize the normal vector of HSVM and feature factor matrices successively. Experiments with two popular HSI datasets verify the effectiveness and superiority of the proposed HSVM.