The ratio of non-performing assets to total assets was 56 basis points, down 1 basis point from the third quarter of 2023 and up 15 basis points from the fourth quarter of 2022. Our non-performing loans to total loans are 47 basis points compared to 46 basis points in the third quarter. In the fourth quarter of 2023, the coverage ratio of loan loss reserves to non-performing loans closed at three times, consistent with three times at the end of the third quarter, an increase from two times at the close of the fourth quarter of 2022. Now moving on to Slide 12, which is a new slide we added this quarter to better show the drivers of the allowance for credit losses. At the end of the fourth quarter, the allowance was $95.5 million, a decrease of $3.3 million or 3.3% compared to $98.8 million at the close of the third quarter.
The drivers of the allowance movement this quarter were $20.6 million in charge-offs, out of which $12.1 million are incremental charges in the quarter and are primarily related to the indirect consumer portfolio, the exit of the New York City nonperforming loan and some smaller balanced business loans and $4.5 million release due to the transfer of the Houston multifamily loans to held for sale. These are offset by $2.6 million related to credit quality and macroeconomic sector updates, $1.8 million due to net loan growth, and $5.3 million in recoveries, primarily related to a commercial LatAm loan that was charged off back in 2017. We recorded a provision for credit losses of $12.5 million in the fourth quarter compared to an $8 million provision in the third quarter.
The provision included $0.5 million for reserves for contingencies. Slide 13 is also new and provides a closer look on this topic to illustrate what the incremental charges in the quarter were excluding previously reserved items. In that line, the $20.6 million in charge-offs included $10.3 million from the CRE New York nonperforming loans that we exited in 4Q for which we had recorded specific reserves of $8.5 million in 3Q. Therefore, the impact for this quarter was only $1.8 million in additional provision expense related to this loan. Additionally, we charged off $7 million related to the indirect purchase consumer loans and $3.3 million due to multiple smaller balanced banking loans. The impact of provision due to these incremental charges was $12.1 million this quarter.
We introduced Slide14 this quarter to provide more color regarding criticized loans. Special mentioned loans increased by $16.4 million or 55.8%. The increase is primarily due to five commercial loans totaling $34.8 million downgraded to special mentioned during the quarter, consisting of one commercial Florida ABL loan totaling $18.7 million, three owner occupied loans totaling $13 million and one commercial Texas unsecured loan totaling $3.1 million. The increases were offset by two commercial loans totaling $17 million that were further downgraded to non-accrual during the quarter as mentioned in the previous NPL discussion. Next, I’ll discuss net interest income and net interest margin on Slide 15. Net interest income for the fourth quarter was $81.7 million, up 4% quarter-over-quarter and down 0.6% year-over-year.
The quarter-over-quarter increase was primarily attributed to higher average rates on total interest earning assets, primarily loans, increased average loan balances and lower average balance in FHLB advances. The increase in net interest income was partially offset by higher average balances embracing money market deposits and customer CDs as well as lower average balances and deposits with banks. Given there were no market rate increases during the quarter, there is no beta calculation for this period. However, we observed a beta of approximately 47 basis points on a cumulative basis since the beginning of the interest rate up cycle as a result of the combined effect of rate increases and transactional deposits, repricing of time deposits that had not repriced at higher rates as well as higher balances in time deposits at higher market rates.
Moving on to the net interest margin. We added Slide 16 this quarter to show the contribution to NIM from each of its components. As Jerry mentioned, NIM for the fourth quarter was 3.72%, up by 15 basis points quarter-over-quarter. The change in the NIM was primarily driven by the increase in the yield of our loan portfolio, which is now at 7.09%, an increase of 32 basis points compared to the third quarter. Interest income for 4Q23 includes $3.6 million in connection with the loan recovery previously charged off, as I mentioned earlier. Excluding the positive impact of this loan, the NIM would be at 3.56%, which is stable when compared with the 3Q23 NIM at 3.57%. The NIM reflects a higher yield of our earning assets offset by the higher cost of funds.
