By the 1920s, the library had outgrown its space. With generous support from Ellen Browning Scripps and others, a new library building was designed by William Templeton Johnson, who would become the architect of many of San Diego's most important civic structures, including Balboa Park's San Diego Museum of Art and Museum of Natural History. The gracious Spanish Renaissance–style building was dedicated and opened to the public in 1921.

For many years, the Library Association, a private institution, supported a general library. In 1955, the City of San Diego agreed to take over the operation of the library as one of its public library branches, and rented the space from the Library Association of La Jolla, or Athenaeum Music & Arts Library, as it became known from then on. The Athenaeum donated the bulk of the library holdings to the public library, retaining certain books on music and art as the nucleus of a collection for the newly named Athenaeum Music & Arts Library.

That same year, artist and architect William Lumpkins was hired to design an adjacent building that would complement the larger library structure. This is the building that houses the Joan & Irwin Jacobs Music Room, and is noted for its striking rotunda—the Athenaeum's "trademark" image. The Athenaeum was dedicated on June 27, 1957.

By 1989, with the Athenaeum's increasing collections and larger membership, the one-room library could no longer serve the needs of the organization and the community. When the general library moved to a new La Jolla branch of the public library on Draper Avenue, the Athenaeum was able to expand into part of the William Templeton Johnson building. La Jolla architect David Raphael Singer designed the expansion, joining the Athenaeum (Lumpkins) building with part of the 1921 (Templeton) library building.

Since the first expansion in July 1990, the Athenaeum has taken on an increasingly active role in the cultural life of San Diego.

The library has added a large number of year-round events and programs, including art exhibitions, concerts (chamber, jazz, and new music), lectures, and art classes at two studio locations. During this time, the Library's membership has doubled, and income from membership, donations, events, and admissions has increased tremendously. In 1999, the Athenaeum celebrated its 100th Anniversary in a whirlwind of activities and publicity that celebrated, as well, its position as one of San Diego's outstanding cultural institutions.

In 2004, an ambitious campaign—the first in the organization's history—was undertaken, to allow the expansion into the remaining portions of the three buildings, which were formerly leased as retail space. The campaign was named “The Campaign to Reclaim,” because the library moved into its own spaces again. In January 2007, the completed expansion and renovation was unveiled with three days of community festivities. The size of the Main Gallery nearly doubled, and the North Reading Room (formerly leased) was opened. The CD and DVD collections were moved to a new area allowing for more space in the Joan & Irwin Jacobs Music Room. The School of the Arts studio was moved into a remodeled area of the building facing Girard Avenue. The Fredman Family Vault was added to house the artists’ books in a climate-controlled environment. And all of the walls, floors, and furniture were restored to their full beauty and elegance.

As one of the founding members of the Membership Libraries Group, the Athenaeum was included in a book, America's Membership Libraries, published in 2007. The book offers the fascinating histories of 16 of the country's remaining membership libraries. It is available to members for checking out from the collection.