Abstract: In order to study stability evolution mechanism of rib pillars during highwall mining, based on catastrophe theory, criteria for instability of coal pillars were derived. Secondly, similar material simulating experiment was conducted to study vertical stress's distribution pattern and evolution as well as their relationship with instability of coal pillars with different widths during coal mining and staged loading. Finally, test results were verified through numerical simulation. The results show that when the ratio of yield zone width to pillar width is greater than 88%, coal pillars will undergo instability failure. During mining process, distribution of vertical stress is in a saddle shape, which ensures a stable state. In the process of staged loading, coal pillars will see shear failure in turn from a small width to a large one, and vertical stress in the middle increases sharply, which is a precursor to a failure. The abrupt inflection point of vertical stress can be taken as a pre-warning criterion for coal pillar instability in engineering practice. It is also found that test results of stability are basically consistent numerical simulation ones, which verifies the rationality and reliability of the experiment results.

Key words: highwall mining, rib pillar, coal pillar instability, catastrophe theory, similar material simulation