Integrated satellite-terrestrial network (ISTN) can provide a continuous service for vehicular users in remote areas with a seamless network coverage. However, considering the difference in the usage costs between satellite and terrestrial networks and the variability of services for latency requirements, it is of great significance to design a cost-effective data offloading decision for reducing network overhead and ensuring task delay requirements. In this paper, we design a cost-effective data offloading mechanism for vehicles in ISTN. The default transmission for remote areas is via the satellite, where the terrestrial networks can offload the data with intermittent coverage in an opportunistic manner due to the vehicle mobility. To model the diversity in service delay requirements, a virtual queue is exploited to capture the residual maximum delay tolerance of each service as time elapses. We formulate the satellite-terrestrial collaborative transmission as a non-linear programming (NLP) problem. To solve the problem, we propose a reinforcement learning (RL)-based data offloading algorithm for real-time decision making. Simulation results show that the RL-based data offloading algorithm reduces the network overhead and outperforms other baseline schemes we proposed.