We expect the margin to be stable through the second quarter. More on NIM in my closing remarks. Moving on to interest rate sensitivity on Slide 17. You can see the asset sensitivity of our balance sheet with 52% of our loans having floating rate structures and 56% repricing within a year. We also continue to execute ALM strategies including hedging interest rate risks as we expect the downward trend in interest rate starting in 2024. As we have said in previous calls, we continue to position our portfolio for a change in rate cycle by incorporating rate floor when originating adjustable loans. We currently have 49% over our adjustable loan portfolio with floor rates. Additionally, you can see here that within the variable-rate loans, 36% are indexed to SOFR.
Our net interest income sensitivity profile remained stable compared to the third quarter. We also include the sensitivity of our AFS portfolio to showcase our positioning to benefit from a rate down scenario. As I’ve done in the past calls during this interest rate cycle, I would like to mention the change, in this case, the improvement in AOCL following expectations for easing monetary policy in 2024. We will continue to actively manage our balance sheet to best position our bank for success in 2024 and beyond. Continuing to Slide 18. Noninterest income in the fourth quarter was $19.6 million, a decrease of $2.3 million from the third quarter, primarily due to lower mortgage banking income, reduction adjustment of $0.7 million in connection with the enhancement of BOLI during the quarter, lower fees on customer deposits in the fourth quarter in connection with the FIS conversions, lower gains on the early termination of FHLB advances, and lower loan level derivative income due to less new swap contracts during the quarter.
Offsetting the decrease in noninterest income were higher cards and trade financing, trade finance servicing fees. We consider $5.7 million of our noninterest income as non-recurring item, a decrease compared to $6.9 million in the third quarter. Core noninterest income was $14 million in the fourth quarter compared to $15 million in the third quarter. Amerant’s assets under management in custody totaled $2.3 billion as of the end of the fourth quarter of $197 million or 9.4% from the end of the third quarter. This increase was primarily driven by increased market valuations following the market value we saw in the fourth quarter. Turning to Slide 19. Fourth quarter noninterest expense was $109.7 million, up $45 million or 70% from the third quarter and up $47 million year-over-year.
We consider $43 million of our expenses this quarter as non-routine expense items as previously mentioned. The quarter-over-quarter increase was primarily due to the following. Previously discussed charge in connection to the transfer of the Houston CRE loan portfolio from loans held for investment to loans held for sale. Higher professional fees of fourth quarter expenses included the recurring expenses for FIS for the full quarter, whereas 3Q ‘23 only included expenses for a portion of September. Higher salaries and severance expenses driven by restructuring of business lines and other restructuring activity. Goodwill impairment due to the consolidation of Amerant Mortgage and dissolution plan of Elant Bank & Trust in Cayman, as well as other expenses in connection with the BOLI restructure.
The increase in noninterest expense was partially offset by lower occupancy and equipment expenses since there were no expenses associated with fresh closures during the quarter. In terms of our team, we ended the quarter with 682 FTEs, out of which 65 are in Amerant mortgage, lower from the 700 we had in the third quarter following strategic reductions in headcount across multiple units. Moving on to Slide 20. We reported fourth quarter diluted loss per share of negative $0.51 on net loss of $17.1 million. We recorded an income tax benefit, which impacted our diluted EPS favorably. As we have mentioned earlier, non-interest expense was higher during the fourth quarter, which resulted in the significant net impact of non-routine items on EPS.
I’ll now give some color of our outlook for the first quarter 2024 and 2024 overall. So in summary on the next slide, we will say the following regarding financial expectations. We expect annual loan growth of approximately 15%, our projected annual deposit growth will match loan growth. We intend to focus on improving the ratio of noninterest-bearing to total deposits. Having new treasury management platform and new digital account opening tool should help in this regard. Our loan to be profit target will remain at 95%. The net interest margin is expected to be stable compared to the normalized 4Q23 results at the 3.50% to 3.60% level in the first half of 2024 and improve over the second half of the year. We expect higher expenses in the first half of 2024 given investment and continued expansion predicting to achieve 60% efficiency in the second half of 2024 as we grow.
We intend to continue executing on prudent capital management balancing between retaining capital for growth and buybacks and dividends to enhance returns. And with that, I pass it back to Jerry for 2024 overview and closing remarks.
